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Executive Summary 


This is the first of three evaluation reports on the effectiveness of a regional train- 
the-trainer strategy to support classroom implementation of the Next Generation Content 
Standards and Objectives (NxGen CSOs). This report focuses on six regional Educator En- 
hancement Academies (EEAs) hosted by the eight regional education service agencies (RE- 
SAs) in the spring of 2013. The EEAs prepared RESA-based NxGen trainers who would 
provide professional development for educators— primarily teachers— in schools across the 
state. Later phases of this study will examine the extensiveness and quality of training of- 
fered by the RESA-based NxGen trainers during the subsequent months (Phase 2), and the 
ultimate impacts of those training experiences on teachers’ instructional practices and stu- 
dent performance (Phase 3). 

The six EEAs focused on the NxGen CSOs for elementary school (Grades 2-3), middle 
school (Grades 6-8), and high school (Grades 10-12). 1 They had durations of 2 to 3 days. The 
WVDE supplied content expert trainers for four of the EEAs. Corwin Press provided trainers 
for the remaining two (see Table 1). 

Table 1. 2013 Regional Educator Enhancement Academy Schedule 


RESAs Dates Location Source of trainers 


RESA3 

April 15-17 

Charleston, WV 

Corwin Press 

RESA7 

April 15-17 

Morgantown, WV 

Corwin Press 

RESA2 

May 17, 18, and 28 

Huntington, WV 

WVDE 

RESAs 5 and 6 

May 20 and 22 

Wheeling, WV 

WVDE 

RESAs 1 and 4 

May 29-31 

Beckley, WV 

WVDE 

RESA8 

June 3-5 

Shepherdstown, WV 

WVDE 


In all, 953 participants were prepared to be RESA-based trainers on NxGen instruc- 
tional shifts. Slightly more than a third attended Corwin-led with remainder attending 
WVDE-led EEAs. Attendance at the academies ranged from 140 to 215. All 55 counties were 
represented, and the largest group of attendees was regular education classroom teachers. 

Methods 

An initial evaluation survey, the Event Survey, was conducted using two different 
methods. At WVDE-led academies, participants filled it out onsite either with paper and 
pencil copies or online. Participants at Corwin-led EEAs were contacted via e-mail and re- 
sponded online. A second survey (Follow-up Survey) was administered to all 953 partici- 
pants online in September 2013; it was intended to collect participants views, after they 
conducted their own training during the summer. 


1 During the two previous school years, the remaining grade levels had been the focus of 
NxGen training provided through centralized Teacher Leadership Institutes (TLIs) conducted by the 
West Virginia Department of Education (WVDE). 
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Results 

The overall response rate for the EEA Event Survey was 78%. Response rates for the 
Corwin-led EEAs were considerably lower (54.4%) than for WVDE-led EEAs (89.6%), prob- 
ably due to the challenges posed by the need to collecting data from them through e-mail 
communications during the summer months. Both groups were surveyed together in Sep- 
tember for the EEA Follow-Up Survey; 62.9% responded overall, with WVDE-led EEA par- 
ticipants’ response rate (61.6%) slightly exceeding Corwin’s (56.4%). 

We approach the discussion of findings from two perspectives: first in relationship to 
a review of the research literature published by the WVDE Office of Research (Hammer, 
2013; see a summary in the Introduction of the full report); and second in response to six 
research questions. 

Findings and Recent Research 

Two major components must be in place for professional development to result in 
greater teacher and student learning: (a) a coherent instructional system, and (b) design fea- 
tures that research has shown to be effective (Hammer, 2013). Each of these major compo- 
nents is discussed below. 

Cobb and Jackson (2011) describe what constitutes a coherent instruction system— 
that is, one where various elements work together to raise student achievement. Elements 
especially relevant to this study include the following: (a) explicit goals for students’ learn- 
ing; (b) a detailed vision of high-quality instruction that specifies particular instructional 
practices that will lead to students’ attainment of the learning goals; (c) instructional materi- 
als and associated tools designed to support teachers’ development of these practices; (d) 
district teacher professional development that focuses on the specific practices, is organized 
around the above materials, and is sustained over time; (e) classroom assessments aligned 
with the goals; (f) school-based professional learning communities; and (g) additional sup- 
ports for struggling students. 

The EEAs focused strongly on the first four of the elements described by Cobb and 
Jackson (2011)— that is, participants spent time learning about the NxGen standards for 
their content areas and grade levels (a, above). They also learned about shifts in instruction- 
al approaches that will be needed to teach to the new standards (b, above) and materials and 
tools to help them implement the new approaches in their classrooms (c, above). Partici- 
pants indicated they greatly valued the time spent learning about the instructional shifts and 
related resources, and would like more of both. As for (d) above, the purpose of the EEAs 
was to prepare RESA-based trainers who could lead professional development for educators 
in schools and districts. The extent to this was a successful effort is the focus of this three- 
phase evaluation. Some preliminary evidence is presented later in this summary. The re- 
maining elements in Cobb and Jackson’s system (e-g, above) were not taken on explicitly in 
the EEAs, but could be the focus of future training. 

Turning now to research-based approaches to professional development, consensus 
has developed around the need to include the following five design elements: (a) content and 
content pedagogy focused; (b) coherence with school and district goals, as well as partici- 
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pants level of knowledge and need for training; (c) active learning including time for plan- 
ning implementation of newly learned practices; (d) collective participation of educators 
from the same school or district; and (e) duration (at least 30 hours) and time span (a year 
or more). 

Evidence from the Phase 1 study supports at least the first three practices being pre- 
sent during the 2- or 3-day EEAs. The academies were strongly focused on content and ap- 
proaches to teaching that content (a, above). Further, participants generally agreed that the 
training was tied to their school and district goals, although a small group of participants 
suggested that the content of the training was a mismatch with their needs (especially grade 
or programmatic level) or that the content had already been covered in previous training (b, 
above). As for active learning (c, above), in open-ended comments participants expressed 
appreciation for the discussions and information sharing that took place, as well as the op- 
portunities for collaboration and planning. Participants in WVDE-led EEAs also indicated 
strong agreement that the EEA they attended included opportunities to practice. Partici- 
pants at both Corwin- and WVDE-led sessions agreed that they had opportunities to collabo- 
rate. Phase 2 of this study may provide evidence of the final two design elements (d and e, 
above). 

Findings in Response to the Research Questions 

This part of the discussion directly addresses six research questions that guided the 
Phase 1 study. In doing so, we must address the differences between the two main provid- 
ers— Corwin and WVDE— because the findings were notably different. 

EQi. To what extent did the EEAs deliver high quality professional development? 
When asked if the session they attended included six different research -based practices, par- 
ticipants in WVDE-led EEAs had mean scores that fell solidly in the agreed or strongly 
agreed range. On the other hand, for three of the six indicators, participants in Corwin-led 
sessions had mean scores in the neutral range, with the other three indicating weak agree- 
ment. 

EQ2. To what extent did the EEAs employ appropriate logistic procedures includ- 
ing the amount of time spent on the professional development? Participants from both 
groups agreed that the trainers adhered to the schedule. However, there was stronger 
agreement that the WVDE-led sessions had clear objectives and were well organized. Phase 
2 will investigate whether the duration of the training met the 30 or more hours called for in 
research. 

EQ3. To what extent did the EEAs prepare attendees to effectively train others re- 
garding the NxGen CSOs? Participants at the WVDE-led trainings were much more likely 
than those at Corwin trainings to indicate that the training had been a good start and they 
were looking forward to training others or that the training had provided everything they 
needed to train— by a margin of about 2.5 to 1. Conversely, attendees at Corwin-led events 
were about 12 times more likely to indicate they did not feel ready to train others. When 
asked about the quality of the training materials they were provided for use in their own 
trainings, there was agreement in both groups that they had received adequate quantities 
and that the materials were high quality and evidence based— although the WVDE-led par- 
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ticipants agreed more strongly. The Corwin-led participants, however, were neutral about 
the materials being relevant to their training needs and useable for their own trainings, 
while WVDE-led participants quite strongly agreed they were relevant and useful. 

EQ4. To what extent did the EEAs build sufficient knowledge of the NxGen CSOs 
and of critical shifts from the previous standards? The knowledge test included in the EEA 
Event Survey showed comparable results for both providers except in one area: Individuals 
who attended WVDE-led elementary-level English/language arts sessions performed signifi- 
cantly better than those who attended Corwin-led sessions. In no case did the average raw 
score correspond to less than 58% correct. However, it is important to understand the in- 
verse as well. In the worst case (i.e., middle school mathematics), the average respondent 
answered 42% of scenario items incorrectly. In the best case (i.e., elementary mathematics), 
the average respondent answered 22% incorrectly. 

EQs- To what extent did the EEAs use qualified and knowledgeable personnel to 
deliver content? On all of the quantitative measures of trainer quality, the WVDE trainers 
received extremely high mean ratings— at least 4.6 on a 5-point scale. Corwin trainers, on the 
other hand scored at least a full point lower. Corwin trainers’ lowest score was for “Trainers 
modeled desired training techniques.” Participants at the WVDE-led academies were almost 
three times more likely to mention the high quality of the presenters. In response to an 
open-ended question about what had been least useful, nearly 13% of participants in the 
Corwin-led academies thought the quality of the trainers was inadequate; about 5% thought 
none of the training was useful; and another 5% thought there was too much marketing of 
materials taking place. 

EQ6. To what extent did the EEAs provide a better experience as regionally based 
academies, compared with the previous state-based professional development models? 
Although just under 40% of both groups thought their experiences had been about the same 
as in previous events, about 56% of WVDE-led EEA attendees thought their EEA experience 
was more useful, compared with 16% for Corwin— a 40-point difference. Conversely, about 
46% of Corwin attendees thought it was less useful than previous events, compared with 5% 
of WVDE attendees who held that view, which is another 40% gap. 

With the exception of the knowledge test items, the differences between the two 
groups was important and consistent across nearly all measures, with WVDE-led EEAs get- 
ting very high marks on nearly every measure, and Corwin receiving notably lower ratings. 
Evidence from comments supports possible explanations for these differences: 

• Lack of preparation on the part of the Corwin trainers— Trainers lacked knowledge 
about NxGen standards and were often unable to answer participants’ questions. 

• Lack of experience in a train-the-trainer setting— Participants’ gave the trainers rel- 
atively low marks for the materials they provided in terms of their usefulness and rel- 
evance, and indicated a lack of modeling of training techniques they would be able to 
use. Further, only about a quarter of the participants in the Corwin sessions were 
looking forward to training others or felt they had everything they needed compared 
with two thirds of the participants in the WVDE-led sessions. 
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• Mismatched expectations between the RESAs who contracted with them and Cor- 
win Press— The Corwin trainers did not seem to understand their role in training 
RESA-based trainers. Participant complaints about trainers’ efforts to sell their 
books suggest they may have viewed the sessions more as a marketing opportunity 
than as a critical component in West Virginia’s implementation of the NxGen CSOs. 

• Duration of the training— The Corwin -led sessions were significantly briefer than the 
WVDE-led sessions; that is, 12-15 hours compared with 24 hours. 

Limitations of the Study 

Participants from the two RESAs that used Corwin Press were surveyed more than 
two months after their training; participants at trainings provided by the remaining RESAs 
were surveyed immediately, onsite— which poses a risk of temporal bias. Response bias may 
also have played a role in the EEA Event Survey. Due largely to the way the survey was ad- 
ministered, about 90% of WVDE-led EEA attendees responded to the survey compared with 
only about 54% of attendees— perhaps the most vocal ones— in the Corwin-led sessions. The 
six knowledge tests were developed by WVDE staff, tested by at least two outside educators 
working in the appropriate grade level and content area, vetted by members of the WVDE 
Office of Assessment and Accountability, and tested using factor analysis to cull out ineffec- 
tive items. The tests were not normed, however, or subjected to rigorous statistical validity 
and reliability testing, so these results should be viewed as indicative and not summative. 

Recommendations 

Based on what we learned in Phase 1, we have the following recommendations (addi- 
tional recommendations will likely follow as we learn more in later phases: 

• Develop additional materials and associated tools to support teachers' use of instruc- 
tional practices to help students meet the new NxGen standards. Consider using this 
development process as a professional learning opportunity for teachers who would 
create and vet new resources. 

• Sustain the commitment to professional learning among the RESA-based trainers 
and the educators whose learning about the NxGen standards and instructional shifts 
they will guide. Thirty contact hours over the course of a year should be the mini- 
mum, more should be provided if at all possible. 

• Settle on standards for professional development, such as the Learning Forward 
standards adopted by the West Virginia Board of Education, or the components and 
qualities outlined in this report. 

• Develop standards for train-the-trainer events that clearly specify what should be 
provided, for example, adaptable PowerPoint presentations, activity descriptions, 
and modeling of effective training practices. 

• Include standards and clear objectives for training in contracts with vendors and 
other providers, and hold them accountable. 
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• Evaluate the success of the training based on the trainers’ effectiveness in meeting 
the standards and objectives. Publish the results to motivate providers to align their 
offerings to state goals, priorities, and standards for professional learning. 
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Introduction 


This is the first of three evaluation reports on the effectiveness of a new regional 
strategy for driving classroom implementation of the Next Generation Content Standards 
and Objectives (NxGen CSOs) through high quality professional development. This report 
focuses on the first phase, regional Educator Enhancement Academies hosted by the eight 
regional education service agencies (RESAs) in the spring of 2013, which using a train-the- 
trainer model prepared RESA-based NxGen trainers to provide professional development 
for educators— primarily teachers— in schools across the state. Later phases of this study will 
examine the extensiveness and quality of training offered by the RESA-based NxGen train- 
ers, and the ultimate impacts of those training experiences on teachers’ instructional prac- 
tice and student performance. 

Background 

For two years, the West Virginia Department of Education (WVDE) conducted state- 
sponsored professional development opportunities, the Teacher Leadership Institutes (TLI), 
to roll out the West Virginia NxGen CSOs to districts and schools. The NxGen CSOs are West 
Virginia’s equivalent to the Common Core State Standards (CCSSs). The TLIs were a resi- 
dency-based professional development opportunity that took place in a central location over 
the course of a full week, with participants receiving 35-40 hours of training. Counties sent 
teams of teacher leaders to participate in TLI and, in a traditional train-the-trainer model, 
these teacher leaders were then responsible for returning to their school districts and shar- 
ing what they had learned with other teachers. In 2011, the TLI focused on kindergarten and 
in 2012 it focused on Grades 1, 4, 5, and 9. 

In response to recommendations in a study, the Education Efficiency Audit of West 
Virginia’s Primary and Secondary Education System, conducted at the request of the 
Governor by a private consulting firm, Public Works (2012), the West Virginia Board of 
Education (WVBE) called for decentralization of such statewide professional development 
offerings. The WVBE called for future professional development opportunities to be 
conducted regionally as part of an effort to diffuse responsibility to local boards of education 
and regional education service agencies (RESAs). Accordingly, the roll-out of the NxGen 
CSOs for the remaining grade levels was reconceptualized and conducted regionally, 
beginning with a series of six Educator Enhancement Academies (EEAs) intended to prepare 
RESA-based trainers to work with educators in schools and districts across the state. See 
Table 2 below for details about the dates and locations of the EEAs. 

The 2013 EEAs focused on the NxGen CSOs for elementary (Grades 2-3), middle 
(Grades 6-8), and high school (Grades 10-12). The six EEAs had a duration of 2 to 3 days as 
opposed to the 5-day residency model used for TLI. The WVDE supplied expert trainers for 
four of the six EEAs. The remaining two (RESAs 3 and 7) worked with a private vendor, 
Corwin Press, to provide expert trainers. 
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Table 2. 2013 Regional EEA Schedule 

RESAs 

Dates 

Location 

RESA 3 

April 15-17 

Charleston, WV 

RESA 7 

April 15-17 

Morgantown, WV 

RESA 2 

May 17, 18, and 28 

Huntington, WV 

RESAs 5 and 6 

May 20 and 22 

Wheeling, WV 

RESAs land 4 

May 29-31 

Beckley, WV 

RESA 8 

June 3-5 

Shepherdstown, WV 


Teachers, district central office staff, administrators, and other educators who would 
serve as RESA-based trainers attended the six EEAs to build their capacity to understand the 
NxGen CSOs in English/language arts (ELA) and mathematics. Once the EEAs concluded, 
these RESA-based trainers were expected to provide additional training on the NxGen CSOs 
to county and school staff during the summer of 2013. A variety of delivery models were ex- 
pected to be used during this professional development depending on local needs, including 
professional development delivered at the school and county levels, or centrally for a whole 
RESA region. Regardless of the locale of the training, a typical train-the-trainer model was 
used, which begins by selecting and preparing RESA-based NxGen trainers (Phase 1), con- 
tinues by NxGen trainers providing professional development to classroom teachers (Phase 
2), and ultimately impacts both teacher practice and student performance (Phase 3), as illus- 
trated in Figure 1. 


Next Generation Content Standards and Objectives (NxGen CSOs) 


Districts/RESAs 
select PD 

facilitators with 
appropriate 
expertise and 
skills to become 

RESA-based 

trainers. 

Corwin and 
WVDE expert 
trainers prepare 
r RESA-based ^ 
trainers to roll 

out NxGen CSOs 
regionally. 

RESA-based 

trainers' 

knowledge, 

► skills, and ' 
motivation to 
implement 
NxGen CSOs 

increase. 



RESA-based 
trainers provide 
local educators 
’ with PD 1 
experiences and 
ongoing support. 


Local educators' 
knowledge, 
skills, and 
* motivation to 
implement 
NxGen CSOs 
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Teacher and 
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reflect 

instructional ' 
shifts called for 
in NxGen CSOs. 


Student 
performance 
improves, as 
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Smarter 
Balanced 
testing. 


Phase 2 


Figure 1. Educator Enhancement Theory of Action 

The Educator Enhancement Academies employed a train-the-trainer model for the delivery of professional 
development regionally through the eight RESAs. The delivery model begins with the selection of local 
educators, who receive training, which increases their ability to train others back in their schools or 
districts, with the ultimate goal of improving teacher practice and student performance. 


The EEA delivery model had six primary objectives: 

1. Deliver high quality professional development that results in significant positive 
changes in teachers’ knowledge, skills and behavior, and attitudes and beliefs. 

2. Employ appropriate logistic procedures including the amount of time spent on 
the professional development. 

3. Prepare attendees to effectively train others regarding the NxGen CSOs. 
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4. Build sufficient knowledge of the NxGen CSOs and of critical shifts from the pre- 
vious standards. 

5. Utilize qualified and knowledgeable personnel to deliver content. 

6. Provide a better experience as regionally based academies, compared with the 
previous state-based professional development model. 

Relevant Scholarship 

Train-the-trainer models for professional development have been used in various 
professions to bring innovations to scale. Research and evaluation studies have provided ev- 
idence, albeit somewhat limited, that such train-the-trainer models can support the devel- 
opment of knowledge and skills, and change practice in professional and community 
settings, such as in the following examples: 

• teaching resilience skills to sergeants and training them in how to teach those skills 
to soldiers (Reivich, Seligman, McBride, 2011); 

• promoting evidence-based practices in the education and treatment of children with 
autism and other disabilities (Bryson et al., 2007; LaVigna, Christian, & Willis, 2005; 
Thomas, 2005); 

• training trainers to work with parents of children with disabilities or behavior chal- 
lenges (Hester, Kaiser, Alpert, and Whiteman, 1995; Kaiser, Hester, Alpert, & 
Whiteman, 1995; Kuhn, Lerman, & Vorndran, 2003); 

• developing first aid and risk-reduction skills among teenagers (Carruth et al., 2010); 
and 

• supporting the adoption of positive behavior supports and behavior management in- 
terventions in schools and classrooms (Marchant, Christensen, Womack, Conley, & 
Fisher, 2010; Noell, Duhon, Gatti, & Connell, 2002). 

Pearce and colleagues (2012) conducted a systematic review of the research literature 
examining train-the-trainer programs in health and social care fields, and identified 18 stud- 
ies that met criteria for inclusion. The train-the-trainer interventions varied greatly in their 
approaches, making it impossible to do a meta-analysis of the findings; however, their narra- 
tive review found that of the 18 train-the-trainer interventions, 13 helped to increase 
knowledge, improve clinical behavior, or produce better patient outcomes. Another three 
found possible positive effects, one found no effects, and another, comparing the effect of the 
train-the-trainer program with the use of a CD-ROM-based program, found the latter pro- 
duced a stronger effect. Looking across the studies, the researchers concluded that there is 
evidence to support using a blended learning approach to deliver training for trainers— that 
is, using techniques such as interactive, multifaceted methods and accompanying learning 
materials— to effectively disseminate and implement guidelines and curricula among health 
and social care professionals. Pearce and colleagues also noted that the study designs varied 
in that some measured impacts on trainers while others examined impacts on those the 
trainers trained. They urged more research on the topic. 
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We were unable to locate a systematic review of train-the-trainer models in educa- 
tion. However, there has been considerable research, including systematic reviews of the lit- 
erature about professional development in elementary-secondary education more generally, 
especially for teachers. A review of the research on teacher professional development identi- 
fied an emerging consensus on important contextual and implementation characteristics 
that can promote or inhibit teachers’ use of new knowledge and skills in their classroom 
practice. 

First, teachers’ professional development does not happen in a vacuum and should 
not be a purely individual pursuit (Desimone, Porter, Garet, Yoon, & Birman, 2002; Des- 
imone, Smith, & Ueno, 2006). Research suggests that professional development is best 
viewed as one component in an overall system that also requires alignment among tests, pol- 
icy, and curriculum. Further, when curriculum for improving teaching overlaps with curricu- 
lum and assessment for students, teaching practice and student learning are more likely to 
improve. On the other hand, when policies and implementation do not meet these condi- 
tions— for example, by introducing new assessments or curriculum without offering teachers 
adequate opportunities to learn them or by offering professional development that is not 
well aligned— the chances for success are greatly reduced (Cobb & Jackson, 2011; Cohen & 
Hill, 1998). 

Within this context, research has shown that effective professional development 
tends to have the following elements: 

• Content and content pedagogy focus — This element includes both deepening teach- 
ers’ knowledge of the subject matter they are teaching and the pedagogical approach- 
es that have been shown to be successful in helping students learn that subject 
matter. Effectiveness is improved if the professional development uses the curricu- 
lum materials that teachers will later use with their students (Blank, de las Alas, & 
Smith, 2008; Carpenter et al., 1989; Clewell et al., 2004; Cohen & Hill, 1998, 2001; 
Desimone, Porter, Garet, Yoon, & Birman, 2002; Desimone, Smith, & Phillips, 2013; 
Doppelt et al., 2009; Garet et al., 2001; Kennedy, 1998; McCutchen et al., 2002; 
Penuel, Fishman, Yagamuchi, & Gallagher, 2007). 

• Coherence — This element involves providing professional development experiences 
in a progression that builds on previous experiences and aligns with school goals and 
with state standards, curriculum, and assessments. Coherent professional develop- 
ment programs encourage continuing professional communication among teachers, 
either in their own school or with others in the district who teach similar subject mat- 
ter or students (Cohen & Hill, 1998; Desimone et al., 2002; Garet et al., 2001; Grant, 
Peterson, & Shojgreen-Downer, 1996; Lieberman & McLaughlin, 1992). 

• Active learning — Opportunities for active learning can include reviewing student 
work, practicing a new skill and obtaining feedback, planning how new curriculum 
materials and new teaching methods will be used in the classroom, and engaging in 
discussions and in written work (Desimone et al., 2002; Garet et al., 2001; Penuel, 
Fishman, Yagamuchi, and Gallagher, 2007). 
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• Collective participation— Professional development that has collective participation 
of teachers from the same school, department, or grade helps increase opportunities 
to discuss concepts, skills, and problems that arise when teachers work to integrate 
what they have learned into their classroom practice (Desimone et al., 2002, p. 102; 
Desimone, Smith, & Ueno, 2006; Johnson, Kahle, & Fargo, 2007; Penuel, Fishman, 
Yagamuchi, & Gallagher, 2007; Saunders, Goldenberg, & Gallimore, 2009). Over 
time, it can lead to a professional culture in which teachers in a school or teachers 
who teach the same grade or subject develop a common understanding of instruc- 
tional goals, methods, problems, and solutions— an understanding that is sustained 
over time, even when some teachers leave and others join the group (Garet et al., 
2001). 

• Duration, including time span and contact hours. Depending on the complexity and 
difficulty of the knowledge and skills teachers are learning, the number of contact 
hours may vary, but research suggests that at least 30 hours are needed to impact 
teacher practice and/or student achievement (Blank et al., 2008; Clewell et al., 2004; 
Yoon, Duncan, Lee, Scarloss, & Shapley, 2007). Sustaining the experience over one 
or more school years is also important, allowing for more opportunity for teachers to 
try out new practices and benefit from additional feedback and communication with 
trainers, coaches, or colleagues in professional learning communities in their schools 
(Johnson, Kahle, & Fargo, 2007; Grigg, Kelly, Gamoran, & Borman, 2013). 

Goals of the Full Evaluation 

This report will examine the first of the three phases illustrated in the EEA logic 
model (Figure 1, page 2); two additional reports will focus on the remaining two phases. 
Briefly, each of the phases will include the following areas of inquiry. 

Phase 1 . Preparation of RESA-based, NxGen trainers. This phase of the study exam- 
ines the initial training provided in the six EEAs hosted by the RESAs during the spring and 
early summer of 2013. We will report (a) the number of RESA-based, NxGen trainers pre- 
pared statewide and by region; (b) the results of a knowledge test that assesses the extent to 
which the newly prepared trainers have the knowledge they need about the critical shifts 
represented by the NxGen CSOs; (c) trainers’ immediate perceptions of the quality of the 
training and materials they received; and (d) their retrospective insights about the same af- 
ter conducting their own training. We will compare WVDE and non-WVDE vendor-led 
trainings to assess any important differences. 

Phase 2 . Training provided by RESA-based, NxGen trainers. In late spring of 2014, 
we will report (a) how many of the RESA-based NxGen trainers followed through by provid- 
ing their own NxGen CSO training and details about how many educators attended, from 
which content areas and programmatic levels; (b) the context of the rollout in the eight re- 
gions, including challenges trainers faced; (c) what additional supports they could use from 
RESAs; (d) their plans for follow up; and (e) the perceptions of teachers about the training 
they received from the RESA-based trainers and the impact of that training on their 
knowledge, practice, and beliefs. 
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Phase p . Teacher practice and student performance. This phase of the study, sched- 
uled for the fall of 2016 will report on data collected from a representative sample of educa- 
tors and administrators statewide about (a) the ongoing presence and quality of training, (b) 
local implementation of NxGen CSOs-based instruction, and (c) external support provided 
to educators. We will create regression models that estimate individual and combined im- 
pacts of these factors on student achievement data. 

Goals of the Phase 1 Study 

This study addresses the following evaluation questions to examine the extent to 
which the new EEA delivery model accomplished six primary objectives outlined above. 

EQi. To what extent did the EEAs deliver high quality professional development? 

EQ2. To what extent did the EEAs employ appropriate logistic procedures including the 
amount of time spent on the professional development? 

EQ3. To what extent did the EEAs prepare attendees to effectively train others regarding 
the NxGen CSOs? 

EQ4 To what extent did the EEAs build sufficient knowledge of the NxGen CSOs and of 
critical shifts from the previous standards? 

EQ5. To what extent did the EEAs use qualified and knowledgeable personnel to deliver 
content? 

EQ6. To what extent did the EEAs provide a better experience as regionally based acade- 
mies, compared with the previous state-based professional development models? 
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Population Characteristics 

Participants in the Educator Enhancement Academies (EEAs) were members of vari- 
ous role groups, including academic coaches, district central office staff, instructional sup- 
port teacher/specialist (non-special education), principals/assistant principals, regular and 
special education teachers, and regional education service agency (RESA) staff. They worked 
in schools, districts, and RESAs across the whole state. 

Sampling Procedures 

Trainers at each of the EEAs maintained sign-in logs of all attendees, which included 
the following information (a) the attendee’s name, (b) the region in which he or she attended 
the EEA, and (c) an email address that could be used for follow-up during the summer 
months and early fall. We used the fall contact list from all six EEAs to comprise our sam- 
pling frame for the surveys. 


Measures and Covariates 

To address the evaluation questions, we used a two-stage survey. We conducted an 
initial survey (Event Survey) at the conclusion of each of the Educator Enhancement Acad- 
emies (EEAs). A second survey (Follow-up Survey) was sent to the same individuals in Sep- 
tember 2013. All survey items were developed by the Office of Research in consultation with 
the Office of Early Learning and the Office of Secondary Learning. The latter two offices 
spearheaded the development of the 12 vignettes and related questions used to measure the 
extent to which EEA attendees finished the academy with sufficient knowledge of the in- 
structional shifts called for in the Next Generation Content Standards and Objectives 
(NxGen CSOs). Staff from the Office of Assessment and Accountability checked the quality of 
the knowledge test items, and all items were pilot tested by at least two individuals working 
in the grade level and content areas targeted. Most of the other items in both surveys were 
derived from previous surveys or were easy-to-interpret, open-ended questions. Survey ad- 
ministration was organized and coordinated by the Office of Research. Both survey instru- 
ments are included in Appendix A (page 35). 

Event Survey 

The initial (event) survey included items to assess each of the six evaluation ques- 
tions noted above. For four of the EEAs— those where West Virginia Department of Educa- 
tion staff provided the training— this survey was administered at the end of the event itself, 
either online or with a paper-and-pencil version, when internet connectivity was limited. We 
provided WVDE staff conducting the training a protocol for data collection at the events. 
Participants in the remaining two EEAs led by trainers from Corwin Press, were invited to 
respond to the survey via an automated SurveyMonkey series of invitations and reminders. 
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The majority of items were attitudinal in nature and assessed participants’ percep- 
tions regarding the professional development using Likert type response options and a small 
number of open-ended questions. Additionally, we developed items to serve as a knowledge 
test. Each respondent was asked to read two short scenarios (approximately too words) re- 
lated to their content area (e.g., middle school English/language arts (ELA) or elementary 
school mathematics) and then respond to a corresponding set of knowledge test items. Re- 
spondents also had the opportunity to provide a brief open-ended response that explained 
their answer choices. Responses to the knowledge test items were scored by the Office of Re- 
search and summaries of right/wrong responses provide an indicator of how effectively the 
EEAs addressed critical concepts. We considered this method to be superior to assessing 
content knowledge using perceptual data. 

Follow-up Survey 

The second (follow-up) survey collected perceptual data from EEA participants, after 
they had trained teachers back in their home districts and schools during the late spring and 
summer. This survey was conducted from September 9-October 7, 2013. Survey items asked 
participants how well prepared they were to deliver the training, how many teachers they 
trained, the duration of their training, any follow-up they have planned, and other related 
questions. Open-ended questions solicited information about what aspects of the EEAs 
proved, in retrospect, to have been the most and least useful to prepare them for their own 
trainings, and what other supports they could use from their RESAs going forward. This re- 
port will summarize findings from only a portion of this survey. The remainder will be in- 
cluded in the Phase 2 report, along with additional data collected in the spring of 2014. 

Research Design 

Table 3 displays the evaluation questions, including which items in the surveys and 
other data sources were used to address each. We used descriptive statistics and qualitative 
methods to analyze data gathered from the surveys. We analyzed survey responses in aggre- 
gate for all EEAs, for each EEA., and in some cases, by provider group— that is, Corwin Press 
and the West Virginia Department of Education (WVDE). 

For the knowledge tests included in the Event Survey, we conducted our analysis of 
EEA scenario data in two phases. Phase 1 included verification of the quality of the items in- 
cluded in each scenario. Specifically, we examined two facets of item quality, difficulty and 
discrimination. We utilized the percentage of correct responses for each item as a measure of 
item difficulty. Our goal was to identify any items that were either too difficult or too easy for 
respondents. We defined “too easy” as any item where more than 85% of participants re- 
sponded correctly and “too difficult” as any item where less than 15% of participants re- 
sponded correctly. Items meeting these criteria were flagged. We next examined item 
discrimination, an estimate of the extent to which a correct response on a given scenario 
item is related to successful performance on the overall scenario instrument. For these anal- 
yses we used the point biserial correlation between test takers’ right/wrong responses for 
each item and their total score for the scenario excluding the item under examination. We 
defined poor item discrimination as a point biserial correlation of .20 or lower. If any item 
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was flagged for both difficulty and poor discrimination, we removed that item from our 
analyses before proceeding to phase two. 

Table 3. Summary of Evaluation Questions and Data Collection Methods Used 


Evaluation question 

Data collection method employed 

EQ1. To what extent did the EEAs deliver high 
quality professional development? 

Delivery of high quality professional development 
• Event Survey items 2. 1.1-2. 1.6. 

EQ2. To what extent did the EEAs employ 

appropriate logistic procedures including the 
amount of time spent on the professional 
development? 

RESAs use of appropriate logistic procedures 

• Event Survey items 2. 1.7-2. 1.9 

RESAs provision of appropriate amount of time 

• Personal communications with trainers and RESA staff 


EQ3. To what extent did the EEAs prepare Attendees are well trained to train others 

attendees to effectively train others regarding • Event Survey item 4.1 


the NxGen CSOs? 

• Follow-up Survey items 2.10-2.12 (qualitative 
feedback) 

Attendees are well supplied/equipped to train others 

• Event Survey items 3. 1.1-3. 1.4 

• Follow-up Survey items 2.10 and 2.12 

EQ4. To what extent did the EEAs build sufficient 
knowledge of the NxGen CSOs and of critical 
shifts from the previous standards? 

Attendees are knowledgeable about critical shifts in 

NxGen CSOs 

• Event Survey scenarios 


EQ5. To what extent did the EEAs use qualified and Trainers were qualified and knowledgeable 
knowledgeable personnel to deliver content? • Event Survey items 3. 2. 1-3. 2. 6 



• Follow-up Survey items 2.10-2.11 

EQ6. To what extent did the EEAs provide a better 
experience as regionally based academies, 
compared with the previous state-based 
professional development models? 

Quality of experience provided by regional versus state- 
based professional development 
• Event Survey item 4.2 


Phase 2 of the knowledge test involved examining respondents’ performance on the 
overall scenarios. For these analyses, we first calculated a total scenario score for each re- 
spondent. This was defined as the sum of correct responses across both scenarios to which 
they responded (hereafter referred to as “raw score”). Any respondent who did not complete 
both scenarios did not receive a raw score. We then analyzed raw scores for all respondents, 
and by training sponsor (WVDE, Corwin/WVDE). We conducted descriptive analyses in- 
cluding the average raw score for participants and percentage of the total possible score rep- 
resented by this number (e.g., percent correct). We also conducted independent samples t 
tests to test for differences in average raw scores by provider. 
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Results 


We begin with response rates and demographic information about respondents to 
the survey, and proceed with findings for each of the research questions, drawing from data 
sources as outlined in Table 3 (page 9). 


Recruitment 

There were six Educator Enhancement Academies (EEAs) held from mid-April 
through early June; all were sponsored by regional education service agencies (RESAs). 
RESAs 3 and 7 hosted the two EEAs held in April, which featured trainers supplied by 
Corwin Press. These RESAs opted to provide lists of email addresses of attendees, which the 
Office of Research used to invite participants to respond to the Event Survey. There was 
considerable lag time, however, before the Office of Research received those lists, so 
attendees at these two events were not contacted until June, extending into July 2013. The 
other six RESAs sponsored the remaining four EEAs, which featured trainers from the West 
Virginia Department of Education (WVDE). The WVDE trainers administered the Event 
Survey onsite at the end of the training, either online or in paper and pencil form, depending 
on the availability of adequate internet connectivity. 


Table 4. 


Number of Participants in Event and Follow-Up Surveys and Response 
Rates by Provider and EEA Event 


Event survey 


Follow-up survey 


Participants Respondents 


Participants 


(N) 


(N) 


Response 
rate (%) 


Respondents 

(N) 


Response 
rate (%) 


The overall 
response rate for 
the EEA Event 
Survey was 78%. 

Response rates for 
the Corwin-led 
EEAs were con- 
siderably lower 
(54.4%) than for 
WVDE-led EEAs 
(89.6%), probably 
due to the chal- 
lenges posed by 
collecting data 
from school and 
district personnel 
during the sum- 
mer, when many 
were away from 
school or vaca- 
tioning (see Table 4 for numbers of respondents and rates of response). This factor must be 
taken into consideration when weighing responses from this group. Both groups were sur- 
veyed together in September for the EEA Follow-Up Survey; 62.9% responded overall, with 
WVDE-led EEA participants’ response rate (61.6%) slightly exceeding Corwin’s (56.4%). Ta- 


By Provider 

All EEAs 

953 

743 

78.0 

599 

62.9 

Corwin-led EEAs 

349 

190 

54.4 

197 

56.4 

WVDE-led EEAs 

604 

541 

89.6 

372 

61.6 

NR* 


12 


30 


By EEA 

All 

953 

743 

78.0 

599 

62.9 

RESAs 1&4 EEA 

215 

182 

84.7 

132 

61.4 

RESA 2 EEA 

140 

140 

100.0 

77 

55.0 

RESA 3 EEA 

140 

75 

53.6 

86 

61.4 

RESAs 5 & 6 EEA 

106 

102 

96.2 

75 

70.8 

RESA 7 EEA 

209 

115 

55.0 

111 

53.1 

RESA 8 EEA 

143 

115 

80.4 

88 

61.5 

NR* 


12 


30 



* Respondents did not indicate which RESA provided their training. 
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ble 4 also displays the breakdown by the six EEA events. In all cases, the majority of partici- 
pants responded to the surveys. 

Statistics and Data Analysis 

This section begins with demographic information about the survey respondents, 
then provides results to the surveys by evaluation question, as outlined in Table 3 (page 9). 

Demographic information about respondents 

In all, 953 individuals were trained in the six EEAs, with slightly more than a third 
attending Corwin-led events and a little less than two thirds attending WVDE-led events. 
The numbers at each of the events ranged from 140 to 215 (see Table 4). 

By far, the largest group of respondents to both surveys was general classroom teach- 
ers, making up about two thirds of both groups of respondents, followed in descending order 
by district central office staff (10%), instructional support teachers/specialists (8%), other 
role groups (6%), and with less than 5% each, principals, special education teachers, and 
RESA staff (Table A - 1, page 43 in the Appendix). Respondents to the EEA Event Survey— 
the survey with the highest response rate (78% overall) and, therefore, the most complete 
picture of the attendees of the EEAs— originated from all 55 counties. So, these findings indi- 
cate that there was good representation across the state at the academies, and that the peo- 
ple being trained were, for the most part, teachers and district central office staff who would 
return to their schools and districts to provide training to their colleagues. 

EQ1. To what extent did the EEAs deliver high quality professional development? 

Six items on the EEA Event Survey— each designed to measure some aspect of what 
research has shown to be effective professional development practice— solicited data about 
participants’ experiences at the academies (see Figure 2 for items). The mean score for all six 
items together make up a composite PD Quality Index score. The PD Quality Index score 
based on all participants at all six academies was high— about 4.2 on a 5-point scales. How- 
ever, when looking at responses of participants at academies where Corwin provided the 
training (AT = 184) versus where WVDE provided it (AT = 531), WVDE-led training scored 
about 1 point higher on the PD Quality Index score (4.4) compared with Corwin (3.4), as 
shown in Figure 2 and in Table A - 2 (page 45). 
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PD Quality Index 
PD was intensive 
PD had opportunities to practice 
PD was content focused 
PD was tied to school/district goals 
PD had opportunities to collaborate 
PD had follow-up/continuous feedback 



1 2 3 4 5 

Mean scale score 


Figure 2. Perceived Quality of Professional Development Provided at the Educator 
Enhancement Academies by Source of Trainers 
Overall, academies with WVDE trainers received higher mean scores than those 
with Corwin trainers on all six measures in the PD Quality Index. Both providers 
scored highest on opportunities to collaborate and on their content focus. Data 
source: EEA Event Survey. 


EQ2. To what extent did the EEAs employ appropriate logistic procedures including the 
amount of time spent on the professional development? 

With regard to logistical aspects of the academies, an overall Logistics Quality Index 
mean score— made up of three items in the EEA Event Survey covering the level of organiza- 
tion, clarity of objectives, and adherence to a schedule— was high at 4.3 on a 5-point scale. 
WVDE again outscored Corwin, although by a narrower margin, 4.5 versus 3.7 respectively 
(see Figure 3 and Table A - 3, page 45 in the Appendix). 

As for the appropriateness of the amount of time, this question can only be partially 
answered in the Phase 1 report, which uses data collected during the spring and early fall of 
2013, before RESAs would have had an opportunity to provide any follow-up training they 
may have planned for the participants trained in their respective EEAs. The duration of the 
EEAs themselves varied, however. The sessions led by Corwin included 12-15 contact hours 
over two days; those led by WVDE included 24 contact hours, over three days. Depending 
upon what we learn about possible follow-up, these numbers may fall short of the minimum 
of 30 hours that research has shown are needed to impact teacher practice and/or student 
achievement (Blank et al., 2008; Clewell et al., 2004; Yoon, Duncan, Lee, Scarloss, & Shap- 
ley, 2007). The EEAs were of shorter duration than the 35 to 40 hours of training provided 
in the summer Teacher Leadership Institutes in previous years. 
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Figure 3. Perceived Quality of Logistics at the Educator Enhancement Academies 
by Source of Trainers 

Overall, academies with WVDE trainers received higher mean scores than those 
with Corwin trainers on the three measures in the Logistics Quality Index. Both 
providers scored highest on opportunities to collaborate and on their content focus. 

Data source: EEA Event Survey. 

EQ3. To what extent did the EEAs prepare attendees to effectively train others regarding 
the NxGen CSOs? 

To examine this question, we looked both at participants’ perceptions about how 
ready they thought they were to train others as a result of their participation in the EEA, and 
at how well supplied and equipped they were to conduct their own trainings, using items in 
the Event Survey. In early fall 2013, after most EEA participants had conducted their own 
training, our Follow-up Survey asked in open-ended questions what aspects of the EEAs had 
been the most and least helpful to prepare them. 

Participants’ views about their own readiness to train others diverged greatly be- 
tween the two groups. Participants at the WVDE-led trainings were much more likely than 
those at Corwin trainings to indicate that the training had been a good start and they were 
looking forward to training others or that the training had provided everything they needed 
to train— by a margin of about 2.5 to 1 (see Figure 4 and Table A - 4, page 45). Conversely, 
attendees at Corwin-led events were about 12 times more likely to indicate they did not feel 
ready to train others. 
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Figure 4. The Extent to Which the Attendees Believed the Educator En- 
hancement Academies Prepared Them to Train Others by Source of 
Trainers 

Nearly two thirds of attendees at WVDE-led EEAs were looking forward to 
training or thought the academy had provided them everything they needed 
to train others. Just over a quarter of attendees at the Corwin-led academies 
felt similarly. On the other hand, a quarter of Corwin-trained attendees 
indicated they did not feel ready to train, compared with only 2% of WVDE-led 
attendees who considered themselves unready. Data source: Event Survey. 


Looking at participants views of materials provided at the academies, there was re- 
markable agreement among the participants at the WVDE-led academies that the materials 
were provided in adequate quantities; were high quality and evidence-based; and were rele- 
vant, and useable for their own upcoming trainings. Participants in the Corwin trainings, 
however, were neutral about the relevance and usability of the materials for their own train- 
ings, but held slightly more favorable views about the quality and supply of the materials 
(see Figure 5 and Table A - 5, page 46). Turning now to responses to open-ended questions 


o > 
p o 

•z Q. 


Available in adequate 


quantities 

High quality, evidence-based 


6 

Relevant to my training needs 

3.1 


Useful/useable for my 

1 1 ,, 


trainings 

1 1 1 


1.0 2.0 3.0 4.0 5.0 


Mean scale score 


■ Corwin 

■ WVDE 


Figure 5. Perceived Adequacy of Training Materials by Source of Trainers 
Participants were asked on the Event Survey to what extent they agree or 
disagree that the materials provided at the EEA had the qualities shown above, 
with 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly 
agree. Data source: EEA Event Survey 
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in the early fall (2013) follow-up survey, participants were asked their views, after conduct- 
ing their own trainings, about what had been the most helpful aspect of the EEA they at- 
tended. Respondents provided 874 comments, which we categorized and rank-ordered by 
the frequency with which each category was mentioned (see summary in Table 5 and full de- 
tails in Table A - 6, page 46). The differences between the two training providers are less no- 
table than in other analyses. Participants most valued the time spent learning about the 
Common Core/Next Generation standards, instructional shifts, mathematic standards, and 
other aspects of using and teaching to the new standards; followed by their introduction to 
various resources (i.e., lessons, examples, resources for classroom use); training materials; 
and discussions and information sharing. One notable difference bears mentioning: Partici- 
pants at the WVDE-led academies were almost three times more likely to mention the high 
quality of the presenters. 

Turning to comments about what was least useful (AT = 422), there were certain activ- 
ities and content considered not useful, especially (in descending order) reading the stand- 
ards/ appendices, the rationale for and history of standards, Smarter Balanced assessment, 
the lectures and reading of PowerPoints (see summary in Table 6 and full details in Table A - 
7, page 48). Some participants also suggested there were useful topics that were missing, es- 
pecially sample lessons, units, and practice information; training guidance, and NxGen 
standards by grade level. A smaller portion made comments suggesting that the content of 
the training was a mismatch with participants’ needs, especially information about pro- 
grammatic level in content areas; others thought the content was redundant with previous 
training. Nearly 13 percent of participants in the Corwin-led academies thought the quality 
of the trainers was inadequate; about 5% thought none of the training was useful and anoth- 
er 5% thought there was too much marketing of materials taking place. Nearly 9% of the 
comments about the WVDE-led academies were complaints about the facilities, especially 
the lack of air conditioning and internet connectivity. 

Table 5. Retrospective View (After Conducting Own Training) of Most Helpful Aspects of EEA: Summary of 


Responses to Open-Ended Question by Source of Trainers 


Category of comments 

Percent of comments 

All Corwin 

WVDE 

Focus on standards and instructional shifts 

26.8 

31.3 

25.2 

Lessons/examples/resources for classroom/training use 

16.5 

13.5 

17.5 

Training materials 

12.2 

10.0 

13.0 

Discussions/information sharing 

9.8 

9.1 

10.1 

Quality of presenters 

7.1 

3.0 

8.5 

Opportunities for collaboration/planning 

5.8 

6.1 

5.7 

Little or nothing 

1.5 

5.2 

0.2 

Miscellaneous (each with <5.0% of total) 

20.3 

21.8 

19.8 


Source: Follow-up survey conducted early fall, 2013. 
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Table 6. Retrospective View (After Conducting Own Training) of Least Helpful Aspects of EEA: Summary of 


Responses to Open-Ended Question by Source of Trainers 


Category of comments 

Percent of comments 

All Corwin 

WVDE 

Specific activity/content not useful 

26.8 

27.3 

26.5 

Specific content missing 

11.4 

13.3 

10.3 

Content mismatched with participants' needs 

7.8 

6.7 

8.5 

Inadequate quality of trainers 

6.2 

12.7 

2.6 

Facilities/food/internet connectivity 

5.7 

0.0 

8.8 

Time wasted/schedule could have been shortened 

5.2 

4.0 

5.9 

None of it was useful 

2.4 

5.3 

0.7 

Too much focus on marketing materials 

2.1 

5.3 

0.4 

All aspects useful 

3.8 

1.3 

5.1 

Miscellaneous (each with <5.0% of total) 

28.6 

24.0 

31.3 


Source: Follow-up survey conducted early fall, 2013. 


EQ4. To what extent did the EEAs build sufficient knowledge of the NxGen CSOs and of 
critical shifts from the previous standards? 

To address this question, we conducted knowledge tests of participants who attended 
all academies. In all there were six tests covering English/language arts (ELA) and mathe- 
matics at all three programmatic levels, elementary (Grades 2-3), middle (Grades 7-8), and 
high school (Grades 10-11). Results of each of the six tests follow. 

Elementary ELA 

For the elementary school ELA scenarios, we found two items did not meet item 
quality standards. As a result, items 5 and 6 of scenario one were removed from subsequent 
analyses (see Table A - 8, page 50). Overall performance on the scenarios was quite good 
with respondents answering 9.91 of 14 items correctly (approximately 71%, see Table 7 be- 
low). When testing for differences among providers, we found that those individuals who 
attended WVDE-led sessions performed significantly better than those who attended Cor- 
win-led sessions. The difference of 2.11 points on average corresponds to an effect size of .62, 
a moderate effect (see Table A - 9, page 50). 

Middle school ELA 

For the middle school ELA scenarios, all items met item quality standards. As a re- 
sult, none were removed from subsequent analyses (see Table A - 10, page 51). Overall per- 
formance on the scenarios was again quite good with respondents answering 7.95 of 11 items 
correctly (approximately 72%, see Table 7 below). When testing for differences among pro- 
viders, we found no significant differences among the scores of those individuals who at- 
tended WVDE-led sessions and those who attended Corwin-led sessions (see Table A - 11, 
page 51). 
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Table 7. Knowledge Test Performance by Provider 


Provider 

Items 

N 

M 

SD 

% Correct 

Elementary ELA scenarios 

ALL 

14 

90 

9.91 

3.01 

70.8 

Corwin 

14 

21 

8.42 

2.74 

60.1 

WVDE 

14 

66 

10.43 

2.92 

74.5 

Middle school ELA scenarios 

ALL 

11 

68 

7.95 

1.88 

72.3 

Corwin 

11 

10 

8.20 

1.75 

74.5 

WVDE 

11 

56 

7.96 

1.91 

72.4 

High school ELA scenarios 

ALL 

11 

78 

7.60 

1.92 

69.1 

Corwin 

11 

34 

7.44 

1.69 

67.6 

WVDE 

11 

44 

7.72 

2.10 

70.2 

Elementary school mathematics scenarios 

ALL 

11 

85 

8.65 

1.35 

78.6 

Corwin 

11 

27 

8.74 

1.43 

79.5 

WVDE 

11 

58 

8.62 

1.32 

78.4 

Middle school mathematics scenarios 

ALL 

9 

115 

5.30 

1.47 

58.9 

Corwin 

9 

9 

5.55 

1.42 

61.7 

WVDE 

9 

105 

5.27 

1.49 

58.6 

High school mathematics scenarios 

ALL 

9 

96 

5.52 

1.20 

61.3 

Corwin 

9 

14 

5.42 

1.08 

60.2 

WVDE 

9 

80 

5.51 

1.20 

61.2 


High school ELA 

For the high school ELA scenarios, all items met item quality standards. As a result, 
none were removed from subsequent analyses (see Table A - 12, page 51). Overall perfor- 
mance on the scenarios was again quite good with respondents answering 7.60 of 11 items 
correctly (approximately 69%, see Table 7 above). When testing for differences among pro- 
viders, we found no significant differences among the scores of those individuals who at- 
tended WVDE-led sessions and those who attended Corwin-led sessions (see Table A - 13, 
page 52). 

Elementary school mathematics 

For the elementary school mathematics scenarios, all items met item quality stand- 
ards. As a result, none were removed from subsequent analyses (see Table A - 14, page 52). 
Overall performance on the scenarios was very good with respondents answering 8.65 of 11 
items correctly (approximately 79%, see Table 7 above). When testing for differences among 
providers, we found no significant differences among the scores of those individuals who at- 
tended WVDE-led sessions and those who attended Corwin-led sessions (see Table A - 15, 
page 52). 
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Middle school mathematics 

For the middle school mathematics scenarios, one item did not meet quality stand- 
ards (i.e., item 3 on scenario 2). As a result, this item was removed from subsequent analyses 
(see Table A - 16, page 53). Overall performance on the scenarios was the lowest of all sce- 
narios. Respondents answered 5.30 of 9 items correctly (approximately 59%, see Table 7 
above). When testing for differences among providers, we found no significant differences 
among the scores of those individuals who attended WVDE-led sessions and those who at- 
tended Corwin-led sessions (see Table A - 17, page 53). 

High school mathematics 

For the high school mathematics scenarios, one item did not meet quality standards 
(i.e., item 3 on scenario 1). As a result, this item was removed from subsequent analyses (see 
Table A - 18, page 53). Overall performance on the scenarios was the second lowest of all 
scenarios. Respondents answered 5.52 of 9 items correctly (approximately 61%, see Table 7 
above). When testing for differences among providers, we found no significant differences 
among the scores of those individuals who attended WVDE-led sessions and those who at- 
tended Corwin-led sessions (see Table A - 19, page 54). 

Summary of Evaluation Question 4 findings 

Figure 6 provides a high level overview of the results presented above. On the surface 
it appears performance was quite good on most scenarios. In fact, in no case did the average 
raw score correspond to less than 58% correct. However, it is important to understand the 
inverse as well. Put another way, in the worst case (i.e., middle school mathematics), the av- 
erage respondent answered 42% of scenario items incorrectly. In the best case (i.e., elemen- 
tary mathematics), the average respondent answered 22% incorrectly. These results indicate 
that there is undoubtedly still work to do in building understanding of the shifts represented 
by the Next Generation Content Standards and Objectives among front line trainers. If these 
individuals are not trained to a sufficient level of expertise, it is unlikely they will be able to 
effectively impart the intended knowledge to teachers and leaders in the field. 
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Figure 6. Overall Findings of Knowledge Tests by Content Area and Program Level 
Based on the knowledge test component of the EEA Event Survey, there is room for 
improvement in the knowledge of trainers who will be providing training in schools 
and districts across the state. 


EQ5. To what extent did the EEAs use qualified and knowledgeable personnel to deliver 
content? 

With mean scores all at 4.6 or above on a 5-point scale, there was strong agreement 
among attendees at WVDE-led EEA sessions that the trainers were knowledgeable, well or- 
ganized, clear, effective presenters, and that they managed discussions well, answered ques- 
tions adequately, and modeled desired training techniques. The ratings for the WVDE 
trainers were the highest received of any of the categories of questions in the Event Survey 
(i.e. use of research-based approaches, appropriate logistic procedures). Participants at the 
Corwin-led events were far less satisfied, with mean ratings that were 0.9 to 1.3 points lower 
on a 5-scale, although still on the positive side of the midpoint 3 (neutral) rating (Figure 7). 

As mentioned earlier (see EQ3 above), in the follow-up survey, many participants 
commented on the quality of the presenters when asked to give a retrospective view (after 
conducting their own training) about what was most useful and least useful in the EEA they 
attended. About 7.1% of attendees at WVDE-led training specifically mentioned the high 
quality of the trainers as having been the most useful aspect, while 3.0% of attendees at 
Corwin-led training made similar comments. On the other hand, nearly 12.7% of attendees 
at Corwin-led training mentioned the inadequate quality of trainers as being the least helpful 
aspect of the EEA they attended, while only about 2.6% of attendees at WVDE-led training 
made similar comments. 
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Figure 7. Attendee Ratings of Trainers by Source of Trainers 

Participants were asked on the Event Survey to what extent they agree or disagree 
that the EEA trainers had the qualities shown above, with 1 = strongly disagree, 2 = 
disagree, 3 = neutral, 4 = agree, and 5 = strongly agree. 

EQ6. To what extent did the EEAs provide a better experience as regionally based 
academies, compared with the previous state-based professional development models? 

The intent of this question was to get a sense of participants’ perceptions about how 
well the RESA-based EEA format compared with previous centralized models offered by the 
WVDE, especially the Teacher Leadership Institutes (TLI). The TLIs for the previous two 
years had also provided training in the Next Generation Content Standards and Objectives 
(NxGen CSOs), although for different programmatic levels. As mentioned earlier, the TLIs 
were a residency-based professional development opportunity that took place over the 
course of a full week, with participants receiving 35-40 hours of training. The EEAs were of 
shorter duration whether offered by Corwin (12-15 hours over two days) or WVDE (24 hours 
over three days). 

The Event Survey included a question that asked, “Which of the following statements 
best describes how the training in this academy compares with other train -the-trainer pro- 
fessional development you have participated in during the past three years?” Response op- 
tions included that the professional development was more useful , about the same, or less 
useful; or two options for not expressing an opinion— that is, no opinion or no basis for 
comparison. Overall about 31% chose the latter responses (i.e., 25% Corwin-led and 33% 
WVDE-led EEAs). Looking only at those who did express an opinion, similar percentages of 
attendees for both providers thought the EEA was about equal to previous train-the-trainer 
events (i.e., 40% for WVDE and 39% for Corwin); however, the similarity stops there. About 
56% of WVDE-led EEA attendees thought their EEA experience was more useful, compared 
with 16% for Corwin— a 40-point difference— while about 46% of Corwin attendees thought 
it was less useful compared with previous events, compared with 5% of WVDE attendees, 
which is another 40% gap (Figure 8). 
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Figure 8. Perceived Usefulness of EEA Professional Development Compared With 
Previous Train-the-Trainer Events by Source of Trainers 
Data in this figure represent only attendees who expressed a view in response to a 
question that compared the EEA they attended to other train-the-trainer events 
they had participated in the previous three years. About 31% of the sample, overall, 
indicated they had no opinion or no basis for comparison. See Table A - 21, page 54, 
for full details. 
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Our discussion of findings in this study will be approached from two perspectives: 
first we will discuss the findings in relationship to a review of the research literature pub- 
lished by the WVDE Office of Research (Hammer, 2013; see a summary in the Introduction 
of this report). Second we will report on findings in response to six research questions we set 
out to answer in our study. 


Findings and Recent Research 

The literature review previously mentioned (Hammer, 2013) found two major com- 
ponents must be in place for professional development to result in greater teacher and stu- 
dent learning: (a) a coherent instructional system, and (b) design features based on research 
and effective practice. Each of these major components is discussed below. 

Professional Learning Within a Coherent Instructional System 

Cobb and Jackson (2011) describe what constitutes a coherent instruction system 
that can lead to improved student achievement. Professional learning is one important ele- 
ment, but its success will be limited if other elements are absent. Those elements are out- 
lined next, along with findings from this study that provide some evidence of the presence or 
absence of each element. 

• Explicit goals for students’ learning (Cobb & Jackson, 2011). With the adoption of 
the NxGen standards, based on the Common Core State Standards, West Virginia 
took a major step forward in developing a more rigorous and coherent system of in- 
struction. 

• A detailed vision of high-quality instruction that specifies particular instructional 
practices that will lead to students’ attainment of the learning goals (Cobb & Jack- 
son, 2011). A major focus of the EEAs was to help the participants learn about the in- 
structional shifts that will be required to teach to the NxGen CSOs in English/ 
language arts and mathematics. There is evidence from the EEA Phase 1 study— 
which included a knowledge test focused on measuring participants’ grasp of the in- 
structional shifts— that there is more work to be done in preparing these front line, 
RESA-based trainers to better understand the instructional shifts themselves, so that 
they, in turn, can guide teachers in their home schools and districts through the chal- 
lenge of employing the new approaches to their daily planning and practice. That 
participants would be receptive to additional training in these areas is evident in 
their comments indicating they most valued the time spent learning about the Com- 
mon Core/Next Generation standards, instructional shifts, mathematic standards, 
and other aspects of using and teaching to the new standards. 

• Instructional materials and associated tools designed to support teachers’ develop- 
ment of these practices (Cobb & Jackson, 2011). After learning about the instruction- 
al shifts, participants indicated they most valued their introduction to various 
resources (i.e., lessons, examples, resources for classroom use); training materials; 
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and discussions and information sharing. Some participants also thought there were 
useful topics in short supply at the training session they attended, including sample 
lessons, units, and practice information; training guidance; and NxGen standards by 
grade level. While WVDE has begun development of a Digital Library of materials, it 
is clear that this element is crucial in the statewide effort to create a coherent instruc- 
tional system based on the NxGen CSOs and that these materials must be used in the 
training participants receive. 

• District teacher professional development that focuses on the specific practices , is 
organized around the above materials, and is sustained over time (Cobb & Jackson, 
2011). The EEAs were the final phase of the roll-out of the NxGen CSOs to all grade 
levels, a rollout that began in 2011 at the Teacher Leadership Institutes (TLI), focus- 
ing first on kindergarten, and then continuing in 2012 with the focus on Grades 1, 4, 
5, and 9. The WVDE then shifted from this centralized approach, which brought in 
teams from schools, to a train-the-trainers approach at the EEAs, which prepared 
RESA-based trainers to work in districts and schools over time. Sustaining this effort 
will be crucial, and will be a focus of the Phase 2 and 3 reports. 

• School-based professional learning communities (PLCs) that provide ongoing op- 
portunities for teachers to discuss, rehearse, and adapt the practices that have been 
introduced in district professional development (Cobb & Jackson, 2011). It is beyond 
the scope of the Phase 1 study to investigate the extent to which PLCs are in place and 
have become an important setting for professional learning leading to the implemen- 
tation of the NxGen CSOs. The WVDE Office of Research does have another study 
underway, however, that will shed light on the level and nature of participation of 
principals and teachers in PLCs statewide. Information from that study will be in- 
cluded in the Phase 2 report. 

The final two elements Cobb and Jackson (2011) identified as being critical to the creation of 
a coherent instructional system include the following: 

• Classroom assessments aligned with the goals for students’ learning that can in- 
form the ongoing improvement of instruction and the identification of students who 
are currently struggling. 

• Additional supports for struggling students to enable them to succeed in main- 
stream classes. 

Although these last two elements were neither mentioned by participants as being topics 
they valued from the EEAs they attended nor valuable topics that were missing, including 
them here fills out the picture, and may point to future directions for RESAs and others to 
consider as priority topics for future professional learning leading to the successful imple- 
ment of the NxGen CSOs. 

Professional Learning Aligned With Research-Based Practices 

As mentioned earlier, our review of the research literature (Hammer, 2013) noted 
that within a coherent instructional system (as outlined above), professional development 
experiences should be structured based on five practices that researchers consistently identi- 
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fy as correlated with improved instructional practice and/or greater student learning. Evi- 
dence from the Phase l study is presented here to provide a sense of trainers’ use of those 
practices during the 2- or 3-day academies. Phase l of this study covers only the academies 
themselves; follow-up activities will be studied as part of Phase 2. 

• Content and content pedagogy focused. The EEAs purpose was to prepare RESA- 
based trainers to train others in their schools and districts on instructional shifts re- 
quired to teach to the NxGen CSOs. Participants generally agreed with the Event 
Survey item that suggested the professional development was content focused. 

• Coherence. This quality includes coherence with participants training needs and with 
school and district goals. A small segment (about 8% overall) made comments sug- 
gesting that the content of the training was a mismatch with participants’ needs, es- 
pecially information pertinent to their programmatic level in content areas; some 
others thought the content was redundant with previous training. Participants gen- 
erally agreed that the training was tied to their school and district goals. 

• Active learning. Open-ended comments revealed that many participants greatly val- 
ued the discussions and information sharing that took place, as well as the opportu- 
nities for collaboration and planning. Participants in WVDE-led EEAs also indicated 
strong agreement that the EEA they attended included opportunities to practice. Par- 
ticipants at both Corwin- and WVDE-led sessions agreed that they had opportunities 
to collaborate. 

• Collective participation. The Phase l study did not collect data about the extent to 
which teams of educators from the same school, content area, or specialization at- 
tended the EEA together; however, the EEAs were intended to help prepare educa- 
tors to collectively implement the NxGen CSOs in their schools and districts. 

• Duration, including time span and contact hours. Depending upon what we learn 
about possible follow-up, the contact hours of professional development that RESA- 
based trainers received— and the time span over which they received them— may fall 
short of the minimum of 30 hours delivered over the course of a year or more, which 
research has shown are needed to impact teacher practice and/or student achieve- 
ment. The duration of the academies ranged from 12 to 24 hours over 2 to 3 days. 

Findings for Research Questions 

In this part of the discussion we will directly address the six research questions that 
guided the Phase 1 study. In doing so, we must address the differences between the two main 
providers— Corwin and WVDE— because the outcomes were notably different. 

EQ1. To what extent did the EEAs deliver high quality professional development? 

Participants in WVDE-led EEAs had mean scores that fell solidly in the agreed or 
strongly agreed range, when asked if the session they attended included six different re- 
search-based practices (i.e., was intensive, included opportunities to practice, was content 
focused, was tied to school/district goals, had opportunities for follow-up discussion and col- 
laboration, and included embedded follow-up and continuous feedback). On the other hand, 
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for three of the six indicators, participants in Corwin-led sessions had mean scores in the 
neutral range, with the other three indicating weak agreement. 

EQ2. To what extent did the EEAs employ appropriate logistic procedures including the amount of 
time spent on the professional development? 

Participants from both groups agreed that the trainers adhered to the schedule. 
However, there was stronger agreement that the WVDE-led sessions had clear objectives 
and were well organized. We will not know until we finish data collection for Phase 2, 
whether the duration of the training met the 30 or more hours called for in research. 

EQ3.To what extent did the EEAs prepare attendees to effectively train others regarding the 
NxGen CSOs? 

Participants at the WVDE-led trainings were much more likely than those at Corwin 
trainings to indicate that the training had been a good start and they were looking forward to 
training others or that the training had provided everything they needed to train— by a mar- 
gin of about 2.5 to 1. Conversely, attendees at Corwin-led events were about 12 times more 
likely to indicate they did not feel ready to train others. When asked about the quality of the 
training materials they were provided for use in their own trainings, there was agreement in 
both groups that they had received adequate quantities and that the materials were high 
quality and evidence-based— although the WVDE-led participants agreed more strongly. The 
Corwin-led participants, however, were neutral about the materials being relevant to their 
training needs and useable for their own trainings, while WVDE-led participants quite 
strongly agreed they were relevant and useful. 

EQ4. To what extent did the EEAs build sufficient knowledge of the NxGen CSOs and of critical 
shifts from the previous standards? 

The knowledge test included in the EEA Event Survey showed comparable results for 
both providers except in one area, individuals who attended WVDE-led elementary-level 
English/language arts sessions performed significantly better than those who attended Cor- 
win-led sessions. In no case did the average raw score correspond to less than 58% correct. 
However, it is important to understand the inverse as well. In the worst case (i.e., middle 
school mathematics), the average respondent answered 42% of scenario items incorrectly. In 
the best case (i.e., elementary mathematics), the average respondent answered 22% incor- 
rectly. These results indicate that there is likely still work to do in building understanding 
among front line, RESA-based trainers of the shifts represented by the NxGen standards. If 
these individuals are not trained to a sufficient level of expertise, it is unlikely they will be 
able to effectively impart the intended knowledge to teachers and leaders in the field. 

EQ5. To what extent did the EEAs use qualified and knowledgeable personnel to deliver content? 

On all of the quantitative measures of trainer quality, the WVDE trainers received ex- 
tremely high mean ratings— at least 4.6 on a 5-point scale. Corwin trainers, on the other 
hand scored at least a full point lower. The lowest score was for Trainers modeled desired 
training techniques, which may help explain why participants in those sessions felt so ill- 
prepared to conduct their own training (see EQ3). In response to open-ended questions 


26 I Educator Enhancement Academies Evaluation Study 


Discussion 


about what they had found most helpful in preparing them to conduct their own training, 
participants at the WVDE-led academies were almost three times more likely to mention the 
high quality of the presenters. In response to an open-ended question about what had been 
least useful, nearly 13 percent of participants in the Corwin-led academies thought the quali- 
ty of the trainers was inadequate; about 5% thought none of the training was useful and an- 
other 5% thought there was too much marketing of materials taking place. 

EQ6. To what extent did the EEAs provide a better experience as regionally based academies, 
compared with the previous state-based professional development models? 

The two groups held starkly different views about how their EEA experience com- 
pared with previous, state-based models. Although just under 40% of both groups thought 
their experiences had been about the same as in previous events, about 56% of WVDE-led 
EEA attendees thought their EEA experience was more useful, compared with 16% for Cor- 
win— a 40-point difference. Conversely, about 46% of Corwin attendees thought it was less 
useful than previous events, compared with 5% of WVDE attendees who held that view, 
which is another 40% gap. 

Possible factors for Corwin Press's comparatively poor performance 

The differences between the two groups was important and consistent across nearly 
all measures, with WVDE-led EEAs getting very high marks on nearly every measure, and 
Corwin receiving notably lower assessments in leading the EEAs for which they were re- 
sponsible. Some possible explanations came to the surface for these differences, including 
the following: 

• Lack of preparation on the part of the Corwin trainers— Evidence to support this 
possibility can be found in multiple participant comments indicating trainers’ lack of 
knowledge about NxGen standards and the inability of trainers to answer many of 
the questions participants asked. 

• Lack of experience in a train-the-trainer setting— Participants’ gave the trainers rel- 
atively low marks for the materials they provided in terms of their usefulness and rel- 
evance, and indicated a lack of modeling of training techniques they would be able to 
use. Ultimately, the strongest argument is in the outcome— only about a quarter of 
the participants in the Corwin sessions were looking forward to training others or felt 
they had everything they needed compared with two thirds of the participants in the 
WVDE-led sessions. 

• Mismatched expectations between the RESAs who contracted with them and Cor- 
win Press— The Corwin trainers did not seem to understand their role in training 
RESA-based trainers. They may have viewed the sessions more as a marketing op- 
portunity than as a critical component in West Virginia’s implementation of the 
NxGen CSOs (note the complaints about trainers’ efforts to sell their books). 

• Duration of the training— The Corwin -led sessions were significantly briefer than the 
WVDE-led sessions; that is, 12-15 hours compared with 24 hours, respectively. If 
Corwin had more time, some of their ratings might have improved. 
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Limitations 

Temporal bias may have played role in the data we collected with the EEA Event Sur- 
vey. The two RESAs that used Corwin Press as the source of their trainers did not administer 
the survey at the end of their trainings as did the RESAs who partnered with WVDE. Fur- 
ther, these RESAs did not supply the e-mail addresses of their attendees to the Office of Re- 
search until more than two months had passed, so participants responded to the survey up 
to three months after they attended the EEAs. 

Response bias may also have played a role in the EEA Event Survey. Due largely to 
the way the survey was administered, about 90% of WVDE-led EEA attendees responded to 
the survey compared with only about 54% of attendees in the Corwin-led sessions. 

The six knowledge tests— for English/language arts and mathematics at all three pro- 
grammatic levels (elementary, middle, and high school)— were developed by staff, tested by 
at least two outside educators working in the appropriate grade level and content area, vet- 
ted by members of the WVDE Office of Assessment and Accountability, and tested using fac- 
tor analysis to cull out ineffective items. The tests were not normed, however, or subjected to 
rigorous statistical validity and reliability testing. Consequently, comparisons cannot be 
made among the different content areas/programmatic levels— that is, a mean rate of 75% 
correct for the middle school mathematics test may not be comparable to a 75% rate for the 
middle school English/language arts. Further, these results should be viewed as indicative 
and not summative. 
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• Continue to develop materials and associated tools to support teachers' development 
of instructional practices to help students meet the new NxGen standards. Consider 
using this development process as a professional learning opportunity for teachers 
who would create and vet new resources. 

• Continue to develop the cadre of RESA-based trainers, by focusing on weak areas in 
the knowledge test results. 

• Sustain the commitment to professional learning about the NxGen standards among 
the RESA-based trainers and the educators whose learning they will guide. Thirty 
contact hours over the course of a year should be the minimum, more should be pro- 
vided if at all possible. 

• Settle on a set of standards for professional development, such as the Learning For- 
ward standards adopted by the West Virginia Board of Education, or the components 
and qualities outlined in this report. 

• Develop standards for train-the-trainer events that clearly specify what should be 
provided, for example, adaptable PowerPoint presentations, activity descriptions, 
and modeling of effective training practices. 

• Include both standards and clear objectives for training in contracts with vendors 
and other providers, and hold them accountable. 

• Evaluate the success of the training based on the trainers’ effectiveness in meeting 
the standards. Publish the results to motivate providers to align their offerings to 
state goals, priorities, and standards for professional learning. 
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Appendix A. Survey Instruments 


Educator Enhancement Academy Regional Event Survey (WVDE-CIS-119) 
Welcome to the 2013 Educator Enhancement Academy Survey 

By filling out this survey, you are agreeing to take part in a research study. The pur- 
pose of the study is to evaluate the effectiveness of the Educator Enhancement Academies. 
What we learn from this study may help improve this program or other professional devel- 
opment programs in the future. Your current participation in the study is limited to complet- 
ing a brief survey, which should not take more than 20 minutes of your time. You will be 
presented with a series of items and asked to indicate your responses by either providing a 
rating or a short written response. 

Taking part in this study will put you at no more risk than you would experience dur- 
ing any normal day. Although you may not benefit directly by taking part in the study, it is 
possible that because of what we learn, the program may improve to better meet your needs 
or the needs of students. Your responses to this survey will be protected and will never be 
revealed as coming from you. All responses will be combined and reported as a group. 

You will receive no monetary or other reward for taking part in this research study. 
Filling out the survey is completely voluntary. If you decide not to take part or to stop at any 
time, there will be no penalties or loss of benefits to you. For more information about the 
Educator Enhancement Academies, you may contact Clayton Burch, executive director of the 
WVDE Office of Early Learning (wburch@access.k12.wv.us) or Carla Williamson, executive 
director of the WVDE Office of Instruction (cljwilli@access.k12.wv.us). Additionally, if you 
have questions about this research study or the survey, you may contact Patricia Hammer, 
coordinator, in the WVDE Office of Research (phammer@access.k12.wv.us). 

This research study has been reviewed and approved by the West Virginia Depart- 
ment of Education (WVDE) Institutional Review Board (IRB). If you want to know more 
about the review of this study, you may contact the WVDE IRB chairperson, Nathaniel 
Hixson (nhixson@access.k12.wv.us). 

You may take this top sheet with you in case you would like to follow up. 

Thank you for taking part in this important effort. 
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PART 1: Tell us about yourself 

1.1. Which Educator Enhancement Academy did you attend? (Check one.) 
O RESA 3 (April 15-17) in Charleston 

O RESA 7 (April 15-17) in Morgantown 
O RESA 2 (May 17, 18, and 28) in Huntington 
O RESAs 5 and 6 (May 20 and 22) in Wheeling 
O RESA 1 and 4 (May 29-31) in Beckley 
O RESA 8 (June 3-5) in Shepherdstown 

1.2. In which district do you currently work? 

1.3. What is your current role? (Check the one role that is the best fit.) 

O RESA staff 

O District central office staff 
O Principal/assistant principal 
O Classroom teacher 
O Special education teacher 

O Instructional support teacher (non-special education) 

O Paraprofessional/aide 
O Other (please specify) 


PART 2: Evaluation Questions— Event Characteristics 

Please indicate to what extent you agree or disagree with the following statements about the professional 
development. 

2.1. The professional development. . . 


Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

2.1.1. was intensive in nature. 

EQ1 

O 

O 

O 

O 

O 

2.1.2. included opportunities to practice new 
skills and receive feedback. 

EQ1 

O 

O 

O 

O 

O 

2.1.3. was specific and content focused. 

EQ1 

O 

O 

O 

O 

O 

2.1.4. was tied to school and district goals for 
student learning. 

EQ1 

O 

O 

O 

O 

O 

2.1.5. included opportunities for collabora- 
tion. 

EQ1 

O 

O 

O 

O 

O 

2.1.6. included embedded follow-up and con- 
tinuous feedback. 

EQ1 

O 

O 

O 

O 

O 

2.1.7. was well organized. 

EQ2 

O 

O 

O 

O 

O 

2.1.8. included clearly stated objectives at the 
beginning of each session. 

EQ2 

O 

O 

O 

O 

O 

2.1.9. followed a clear schedule with sessions 
beginning and ending on time. 

EQ2 

O 

O 

O 

O 

O 
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PART 3: Evaluation Questions— Quality of Materials and Facilitators 

Please indicate to what extent you agree or disagree with the following statements about materials, resources, 


and facilitators at the academy. 

3.1. Materials and resources provided are/were. . . 


Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

3.1.1. available in adequate quantities. 

EQ3 

o 

o 

o 

o 

o 

3.1.2. relevant to my training needs back in my 
home school or district. 

EQ3 

o 

o 

o 

o 

o 

3.1.3. high quality and based on recent research 
and evidence. 

EQ3 

o 

o 

o 

o 

o 

3.1.4. useful and useable for my upcoming train- 
ings. 

EQ3 

o 

o 

o 

o 

o 

3.2. At the academy 1 attended, facilitators. . . 


Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

3.2.1. were knowledgeable about the topic. 

EQ5 

o 

o 

o 

o 

o 

3.2.2. were well organized. 

EQ5 

o 

o 

o 

o 

o 

3.2.3. presented the material clearly and effective- 
ly- 

EQ5 

o 

o 

o 

o 

o 

3.2.4. managed discussions well. 

EQ5 

o 

o 

o 

o 

o 

3.2.5. answered questions raised during the ses- 
sion adequately. 

EQ5 

o 

o 

o 

o 

o 

3.2.6. modeled the training techniques they en- 
couraged us to use in our own training. 

EQ5 

o 

o 

o 

o 

o 

PART 4: Evaluation Questions— Event Usefulness 


EQ3 4.1. Which of the following statements best describes the usefulness of the academy in preparing 

you to train other teachers? (Check one.) 


O It was a good start. 

O It was a good start, but I have a lot of questions. 

O It was a good start, and I look forward to using what I learned to train other teach- 
ers. 

O It provided everything I will need to use when I train other teachers. 

O I don’t think these ideas will work very well in my school or district. 

O It’s too early to tell. 

O I don’t feel ready to train other teachers at this point. (If you checked this box, 
briefly describe what you think you would need to be ready.) 

EQ6 4.2. Which of the following statements best describes how the training in this academy compares 

with other train-the-trainer professional development you have participated in during the past 
three years? (Check one.) 

O The professional development was MORE USEFUL. 

O The professional development was ABOUT THE SAME. 

O The professional development was LESS USEFUL. 

O I don’t have an opinion. 

O I haven’t participated in any other train-the-trainer professional development in 
the past three years. 
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EQ1 

4.3. Which three aspects of the academy were MOST useful? (Briefly describe.) 

4.3.1. 

EQ1 

4.3.2. 

EQ1 

4.3.3. 

EQ1 

4.4. Which three aspects of the academy were LEAST useful? (Briefly describe.) 

4.4.1. 

EQ1 

4.4.2. 

EQ1 

4.4.3. 

EQ3 

4.5. About which three topics (if any) do you need more information or assistance to conduct your 
own training? (Briefly describe.) 

4.5.1. 

EQ3 

4.5.2. 

EQ3 

4.5.3. 

PART 5. Evaluation Questions: Scenarios 


5. For which programmatic level and content area will you be training teachers? 

O English/language arts at the primary/elementary school level 

O English/language arts at the middle school level 

O English/language arts at the high school level 

O Mathematics at the primary/elementary school level 

O Mathematics at the middle school level 

O Mathematics at the high school level 

O I will not be training teachers (Participants choosing this option do not need to re- 
spond to the scenarios.) 


38 I Educator Enhancement Academies Evaluation Study 


Appendix A. Survey Instruments 


Educator Enhancement Academy Follow-Up Survey 

Welcome to the 2013 Educator Enhancement Academy Follow-Up Survey 

By filling out this survey, you are agreeing to take part in a research study. The pur- 
pose of the study is to evaluate the effectiveness of the Educator Enhancement Academies. 
What we learn from this study may help improve this program or other professional devel- 
opment programs in the future. Your current participation in the study is limited to complet- 
ing a brief survey, which should not take more than 10-15 minutes of your time. You will be 
presented with a series of items and asked to indicate your responses by either providing a 
rating or a short written response. 

Taking part in this study will put you at no more risk than you would experience dur- 
ing any normal day. Although you may not benefit directly by taking part in the study, it is 
possible that because of what we learn, the program may improve to better meet your needs 
or the needs of students. Your responses to this survey will be protected and will never be 
revealed as coming from you. All responses will be combined and reported as a group. 

You will receive no monetary or other reward for taking part in this research study. 
Filling out the survey is completely voluntary. If you decide not to take part or to stop at any 
time, there will be no penalties or loss of benefits to you. For more information about the 
Educator Enhancement Academies, you may contact Clayton Burch, executive director of the 
WVDE Office of Early Learning (wburch@access.k12.wv.us) or DeWayne Duncan, executive 
director of the WVDE Office of Secondary Learning (ddduncan@access.k12.wv.us). Addi- 
tionally, if you have questions about this research study or the survey, you may contact Pa- 
tricia Hammer, coordinator, in the WVDE Office of Research (phammer@access.k12.wv.us). 

This research study has been reviewed and approved by the West Virginia Depart- 
ment of Education (WVDE) Institutional Review Board (IRB). If you want to know more 
about the review of this study (IRB-WVDE-014), you may contact the WVDE IRB chairper- 
son, Nathaniel Hixson (nhixson@access.k12.wv.us). 

Thank you for taking part in this important effort. 
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PART 1: Tell us about yourself 

1.1. What is your current role? 

□ RESA staff 

□ District central office staff 

□ Principal/assistant principal 

□ General classroom teacher 

□ Special education teacher 

□ Instructional support teacher/specialist (non-special education) 

□ Other 

1.2. In which context did you attend the Educator Enhancement Academy (Select the one that 
is the best fit.) 

□ As a member of a training cadre that will provide support for implementing Next Genera- 
tion Content Standards and Objectives (NxGen CSOs) ACROSS THE REGION. 

□ As a member of a training cadre that will provide support for implementing NxGen CSOs 
ACROSS MY DISTRICT. 

□ As a school staff member who will provide training and support for implementing NxGen 
CSOs IN MY SCHOOL. 

□ Other [please describe] 

1.3. Have you conducted trainings in your school or district based on the NxGen CSOs since 
attending the Educator Enhancement Academy? 

□ Yes [SKIP to PART 2] □ No [SKIP To PART lA] 

PART lA 

1.4. Please provide information about what prevented you from providing training. 


1.5. What other supports would have made it possible for you to provide trainings, had you 
received them? 


[Exit survey] 
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PART 2: Evaluation Questions 

2.1 In which district(s) did you hold training(s)? 

[drop-down list of school districts] 

[drop-down list of school districts] 

[drop-down list of school districts] 

Others (please list in box below) 

[insert text box] 

2.2. Which of the following role groups attended your training? Check all that apply. 

□ RESA staff 

□ District central office staff 

□ Principal/assistant principal 

□ Regular classroom teacher 

□ Special education teacher 

□ Instructional support teacher/specialist (non-special education) 

□ Other 

2.3. For which programmatic level(s) and content area(s) did you train individuals? Check all 
that apply. 

□ English/language arts at the primary/elementary school level 

□ English/language arts at the middle school level 

□ English/language arts at the high school level 

□ Mathematics at the primary/elementary school level 

□ Mathematics at the middle school level 

□ Mathematics at the high school level 

2.4. In all, how many individuals attended trainings that you facilitated? [insert number box] 

2.5. How many training events did you facilitate? [insert number box] 

2.6. What was the duration of the training event you provided (in hours)? (If you 
facilitated more than one event, indicate the typical [average] duration), [insert number box] 

2.7. Have you or do you plan to provide additional follow-up training or support for the indi- 
viduals you trained this summer? □ Yes □ No 

2.7.1 If yes, in what form? (Check all that apply.) 

□ Additional face-to-face trainings. 

□ Webinars or other online training. 

□ Coaching or mentoring 

□ Other [please describe] 


2.7.2. How many additional hours of training or support do you anticipate you will provide 
during the school year to a typical participant who attended your initial summer training? [insert 
number box] 

2.8. What barriers (if any) are you finding most challenging, in providing your own training? 
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2.9. What specific areas/topics from the NxGen CSOs were most difficult for participants to 
understand? 


2.10. In retrospect, which three aspects of the academy were MOST useful in preparing you to 
train others in your school or district? (Briefly describe) 

1. 

2. 

3- 

2.11. Which three aspects of the academy were LEAST useful? (Briefly describe) 

1. 

2. 

3- 

2.12. About which three topics (if any) could you have used more information or assistance to 
conduct your own training? (Briefly describe) 

1. 

2. 

3- 

2.13.1. Are there others ways that your RESA could help you in your work with educators re- 
lated to the NxGen CSOs this school year? □ Yes □ No 

2.13.2 If yes, please describe: 



[Exit Survey] 
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Table A - 1. Role Groups of Survey Respondents 


Overall 


Event survey 

Follow-up survey 

Role group 

N 

% 

N 

% 

N 

% 

All 

1,342 

100.0 

743 

100.0 

599 

100.0 

General classroom teacher 

889 

66.2 

508 

68.4 

381 

63.6 

District central office staff 

132 

9.8 

68 

9.2 

64 

10.7 

Instructional support teacher/specialist 
(non-special education) 

108 

8.0 

44 

5.9 

64 

10.7 

Other 

81 

6.0 

56 

7.5 

25 

4.2 

Principal/assistant principal 

56 

4.2 

29 

3.9 

27 

4.5 

Special education teacher 

55 

4.1 

27 

3.6 

28 

4.7 

RESA staff 

17 

1.3 

8 

1.1 

9 

1.5 

No response 

4 

0.3 

3 

0.4 

1 

0.2 
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Appendix C. Results of EEA Event Survey and Follow-up 
Survey 


Table A - 2. The Extent to Which the Educator Enhancement Academies Delivered High Quality Professional 


Development by Source of Trainers 


All 


Corwin 

WVDE 

Number 

responses 

Mean 

response 

Number 

responses 

Mean 

response 

Number 

responses 

Mean 

response 

PD Quality Index Score 


4.18 


3.40 


4.44 

PD was intensive 

715 

3.98 

184 

3.19 

531 

4.25 

PD had opportunities to practice 

713 

4.11 

183 

3.23 

530 

4.41 

PD was content focused 

714 

4.31 

184 

3.60 

530 

4.55 

PD was tied to school/district goals 

713 

4.22 

183 

3.56 

530 

4.45 

PD had opportunities to collaborate 

708 

4.46 

181 

3.87 

527 

4.66 

PD had follow-up 

709 

3.99 

183 

2.98 

526 

4.34 

NOTE: Data in this table were collected in the EEA Event Survey. 


Table A - 3. The Extent to Which the Educator Enhancement Academies Used Appropriate Logistic 


Procedures by Source of Trainers 



All 


Corwin 

WVDE 

Number 

responses 

Mean 

response 

Number 

responses 

Mean 

response 

Number 

responses 

Mean 

response 

Logistics quality index 


4.28 


3.71 


4.47 

PD was well organized 

712 

4.21 

183 

3.46 

529 

4.47 

PD had clear objectives 

710 

4.22 

182 

3.64 

528 

4.42 

PD adhered to schedule 

712 

4.41 

182 

4.03 

530 

4.54 

NOTE: Data in this table were collected in the EEA Event Survey. 


Table A - 4. The Extent to Which the Educator Enhancement Academies Prepared Attendees to Train Others 


by Source of Trainers 



All 


Corwin 


WVDE 

Response options 

Number 

responses 

Percent 

Number 

responses 

Percent 

Number 

responses 

Percent 

All responses 

682 

100.0 

178 

100.0 

504 

100.0 

Good start 

98 

14.4 

31 

17.4 

67 

13.3 

Good start, but have lots of questions 

110 

16.1 

35 

19.7 

75 

14.9 

Good start, look forward to training 
others 

303 

44.4 

37 

20.8 

266 

52.8 

It provided everything needed to train 

73 

10.7 

8 

4.5 

65 

12.9 

Doubt these ideas will work well for us 

17 

2.5 

11 

6.2 

6 

1.2 

Too early to tell 

27 

4.0 

12 

6.7 

15 

3.0 

Don't feel ready to train others 

54 

7.9 

44 

24.7 

10 

2.0 

NOTE: Data in this table were collected in the EEA Event Survey. 
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Table A - 5. The Extent to Which the Educator Enhancement Academies Provided Adequate Training 


Materials by Source of Trainers 

Materials resources provided were . . 

All 


Corwin 

WVDE 

Number 

responses 

Mean 

response 

Number 

responses 

Mean 

response 

Number 

responses 

Mean 

response 

Available in adequate quantities 

700 

4.12 

181 

3.64 

519 

4.29 

Relevant to my training needs 

701 

4.14 

181 

3.12 

520 

4.49 

High quality, evidence-based 

699 

4.29 

181 

3.59 

518 

4.53 

Useful/useable for my trainings 

698 

4.17 

181 

3.12 

517 

4.53 

NOTE: Data in this table were collected in the EEA Event Survey 


Table A - 6. Detailed Summary of Retrospective Comments About Most Useful Aspects of EEA by Source of 


Trainer 





Corwin-trained WVDE-trained 



All respondents 

respondents 

respondents 



Subcat. Category 


Subcat. Category 

Subcat. Category 


Category comment 

(. N )* 

N 

% 

( N )* 

N 

% ( N )* 

N 

% 

Total 


874 

100.0 


230 

100.0 

644 100.0 

Specific content 


234 

26.8 


72 

31.3 

162 

25.2 

• Learning about NxGen 

(78) 



(33) 


(45) 



CSOs/ CCSS 
• Participating in 

(53) 



(11) 


(42) 



Instructional practice 
activities 









• Learning about strategies 

(32) 



(6) 


(26) 



for implementation 
• Learning about 

(24) 



(8) 


(16) 



Instructional shifts 









• Learning about Smarter 

(23) 



(4) 


(19) 



Balance/assessment 
• Learning about Standards 

(15) 



(7) 


(8) 



for Mathematics Practice 









• Nonspecific 

(5) 



(2) 


(3) 



• Examining the Appendices 

(4) 



(1) 


(3) 



Training materials 


107 

12.2 


23 

10.0 

84 

13.0 

• PowerPoints 

(65) 



(8) 


(57) 



• Nonspecific 

(16) 



(6) 


(10) 



• Videos 

(15) 



(8) 


(7) 



• Online/electronic 

(11) 



(1) 


(10) 



Discussions/information sharing 


86 

9.8 


21 

9.1 

65 

10.1 

Resources 


83 

9.5 


19 

8.3 

64 

9.9 

• Online/electronic 

(37) 



(7) 


(30) 



• Nonspecific 

(26) 



(4) 


(22) 



• Handouts/booklets/binders 

(20) 



(8) 


(12) 



Table A - 6 continues on next page 
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Appendix C. Results of EEA Event Survey and Follow-up Survey 


Table A - 6. Detailed Summary of Retrospective Comments About Most Useful Aspects of EEA by Source of 
Trainer 


Category comment 


All respondents 
Subcat. Category 
(N)* N 


% 


Corwin-trained 
respondents 
Subcat. Category 
(N)* N 


WVDE-trained 

respondents 


% 


Quality of presenters 


62 

7.1 

7 

3.0 

Lessons/examples/resources for 


61 

7.0 

12 

5.2 

classroom use 






• Nonspecific 

(46) 



(10) 


• Teach 21 

(15) 



(2) 


Opportunities for collaboration/ 


51 

5.8 

14 

6.1 

planning 






Activities 


40 

4.6 

13 

5.7 

• Nonspecific 

(20) 



(9) 


• Hands-on 

(18) 



( 3 ) 


• Group 

(2) 



(1) 


Other non-EEA experiences 


37 

4.2 

15 

6.5 

• Personal experience/ 

(10) 



(2) 


reading/research 






• District training/support 

(8) 



(4) 


• Other RESA 

(6) 



(0) 


training/support 






• TLI 

(5) 



(4) 


• Coaching class 

( 3 ) 



(2) 


• Graduate coursework 

(2) 



(2) 


• Other WVDE training 

(2) 



(1) 


• Nonspecific 

(1) 



(0) 


Training guidance-Modeling 


35 

4.0 

6 

2.6 

strategies/activities 






Miscellaneous 


30 

3.4 

9 

3.9 

Little or nothing 


13 

1.5 

12 

5.2 

Time factors/Scheduling- Count 


9 

1.0 

3 

1.3 

Time to explore/learn- Count 


8 

0.9 

0 

0.0 

Proximity to home/local context 


7 

0.8 

0 

0.0 

All aspects 


4 

0.5 

3 

1.3 

Positive attitudes of 


4 

0.5 

1 

0.4 

presenters/participants 






Participating with colleagues 


3 

0.3 

0 

0.0 


Subcat. Category 

(N)* N % 



55 

8.5 


49 

7.6 

(36) 

(13) 


37 

5.7 


27 

4.2 

(11) 

(15) 

(1) 


22 

3.4 

(8) 



(4) 

(6) 



(1) 

(1) 



(1) 

(1) 




29 

4.5 


21 

3.3 


1 

0.2 


6 

0.9 


8 

1.2 


7 

1.1 


1 

0.2 


3 

0.5 


3 

0.5 


Subc. (N )= Comment subcategory frequency 

NOTE: Data in this table were collected in the EEA Follow-up Survey 
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Appendix C. Results ofEEA Event Survey and Follow-up Survey 


Table A - 7. Detailed Summary of Retrospective Comments About Least Useful Aspects of EEA by Source of 


Trainer 




All 


Corwin 

WVDE 


Subcat. Category 
(N)* N 

Subcat. Category 
% (N)* N 

Subcat. Category 
% (N)* N 

% 

Total 


422 100.0 

150 

100.0 

272 

100.0 

Specific activity/content not 


113 

26.8 

40 

27.3 

72 

26.5 

useful 








• Reading standards/ 

(25) 


(9) 


(16) 



appendices 








• Miscellaneous/nonspecific 

(19) 


(5) 


(14) 



• Rationale for/history of 

(11) 


(10) 


(1) 



standards 








• Smarter Balanced assessment 

(10) 


(2) 


(8) 



• Mathematics activities 

(8) 


(2) 


(6) 



• Lecture 

(8) 


(6) 


(2) 



• Large groups 

(7) 


( 3 ) 


(4) 



• Reading PowerPoints 

(7) 


( 3 ) 


(4) 



• Career-technical instruction 

(6) 




(6) 



• Ice breakers 

(4) 




(4) 



• Guidance on training 

( 3 ) 




( 3 ) 



• Miscellaneous/nonspecific 

(2) 




(2) 



• Drug prevention training 

(2) 




(2) 



Specific content missing 


48 

11.4 

20 

13.3 

28 

10.3 

• Sample lessons/units/ 

(27) 


(10) 


(17) 



practice information 








• Lack of training guidance 

(9) 


( 3 ) 


(6) 



• NxGen standards by grade 

(6) 


(5) 


(1) 



level 








• Elementary math/RLA 

(4) 


(1) 


( 3 ) 



• Miscellaneous/nonspecific 

(1) 


(1) 





• Smarter Balanced assessment 

(1) 




(1) 



Miscellaneous 


39 

9.2 

11 

7.3 

28 

10.3 

Content mismatched with 


33 

7.8 

10 

6.7 

23 

8.5 

participants' needs 








• Content area/ programmatic 

(15) 


(5) 


(10) 



level needs 








• Redundant 

(14) 


(3) 


(11) 



• Miscellaneous/nonspecific 

(2) 


(1) 


(1) 



• Over their heads 

(2) 


(1) 


(1) 



Comment meaning unclear 


28 

6.6 

8 

5.3 

20 

7.4 

Quality of trainers 


26 

6.2 

19 

12.7 

7 

2.6 

Facilities/food/internet 


24 

5.7 



24 

8.8 

connectivity 









Table A - 7 continues on next page 
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Appendix C. Results of EEA Event Survey and Follow-up Survey 


Table A - 7. Detailed Summary of Retrospective Comments About Least Useful Aspects of EEA by Source of 
Trainer 


WVDE 

Subcat. Category 

(N)* N % 

16 

5.9 

14 

5.1 

13 

4.8 

5 

1.8 

2 

0.7 

1 

0.4 

8 

2.9 

7 

2.6 

0 

0.0 

4 

1.5 


All 


Corwin 


Subcat. Category 


(NV 


Time wasted/schedule could have 

been shortened 

All aspects useful 

Too much content for time 

allotted 

Lack of discussion/sharing/ 

planning time 

None of it was useful 

Too much focus on marketing 

materials 

Participant recruitment not 
adequate 

Scheduling of training 
Unclear distinctions between 
NxGen and CCSS 
Lack of resources/materials to 
take away 


N 


16 

14 


10 

9 


Subcat. Category 


% 


(. N )* 


22 5.2 


3.8 

3.3 


12 2.8 


2.4 

2.1 


8 1.9 


1.9 

1.7 

1.4 


N 


% 


4.0 

1.3 
0.7 

4.7 

5.3 

5.3 

0.0 

0.7 

4.0 

1.3 


Subc. (N )= Comment subcategory frequency 

NOTE: Data in this table were collected in the EEA Follow-up Survey 
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Appendix C. Results of EEA Event Survey and Follow-up Survey 


Table A - 8. Knowledge Test Item Statistics— Elementary ELA Scenarios 

% correct PBS correlation Discrimination 


Item Scenario Number tested (difficulty) (discrimination) Difficulty flag 1 flag 2 


1 

1 

101 

85.1 

.124 

Y Y 

2 

1 

101 

42.6 

.151 

Y 

3 

1 

101 

91.1 

.250 

Y 

4 

1 

101 

41.6 

.155 

Y 

5 

1 

101 

94.1 

.113 

Y Y 

6 

1 

101 

96.0 

.126 

Y Y 

7 

1 

101 

86.1 

.265 

Y 

8 

1 

101 

66.3 

.410 


1 

2 

92 

80.4 

.877 


2 

2 

92 

82.6 

.859 


3 

2 

92 

84.8 

.565 


4 

2 

92 

80.4 

.877 


5 

2 

92 

78.3 

.846 


6 

2 

92 

81.5 

.744 


7 

2 

92 

41.3 

.125 

Y 

8 

2 

92 

77.2 

.743 


9 

2 

92 

58.7 

.469 


1 % Correct is < 15% or > 85% 





2 PBS <20 






NOTE: Data in this table were collected in the EEA Event Survey 



Table A -9. 

Tests of Significance in Knowledge Test Score Differences by Provider— 

■Elementary ELA 


Scenarios 






Homogeneity 



Mean 


Variable 

of variance 

T 

DF SIG 

difference 

ES Interpretation 

Raw score 

.871 

2.786 

85 .007 

2.01 

.60 Moderate 

S1S2 clean 





effect- WVDE 


NOTE: Data in this table were collected in the EEA Event Survey 
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Appendix C. Results of EEA Event Survey and Follow-up Survey 


Table A - 10. Knowledge Test Item Statistics — Middle School ELA Scenarios 

% Correct PBS Correlation Discrimination 


Item Scenario Number tested (difficulty) (discrimination) Difficulty flag 1 flag 2 


1 

1 

68 

58.8 

.217 



2 

1 

68 

88.2 

.234 

Y 


3 

1 

68 

70.6 

.249 



4 

1 

68 

58.8 

.428 



5 

1 

68 

72.1 

.339 



1 

2 

69 

72.5 

.321 



2 

2 

69 

81.2 

.152 


Y 

3 

2 

69 

71.0 

.233 



4 

2 

69 

76.8 

.048 


Y 

5 

2 

69 

53.6 

.263 



6 

2 

69 

92.8 

.274 

Y 


1 % Correct is < 15% or > 85% 






2 PBS <20 







NOTE: Data in this table were collected in the EEA Event Survey 




Table A -11. 

Tests of Significance in Knowledge Test Score Differences by Provider— Middle School ELA 


Scenarios 







Homogeneity 






Variable 

of variance 

T 

DF SIG 

Mean diff 

ES Interpretation 

Raw score 

.985 

.363 

64 .718 


-.235 

.09 Small effect - 

S1S2 






no difference 

NOTE: Data in this table were collected in the EEA Event Survey 

Table A -12. 

Knowledge Test Item Statistics 

—High School ELA Scenarios 






% Correct PBS Correlation 


Discrimination 

Item Scenario Number tested 

(difficulty) (discrimination) 

Difficulty flag 1 

flag 2 

1 

1 

79 

54.4 

.134 


Y 

2 

1 

79 

79.7 

.247 



3 

1 

79 

63.3 

.217 



4 

1 

79 

60.8 

.524 



5 

1 

79 

69.6 

.126 


Y 

1 

2 

78 

66.7 

.083 


Y 

2 

2 

78 

71.8 

.196 


Y 

3 

2 

78 

69.2 

.147 


Y 

4 

2 

78 

80.8 

.339 



5 

2 

78 

59.0 

.243 



6 

2 

78 

84.6 

.323 




1 % Correct is < 15% or > 85% 

2 PBS <20 

NOTE: Data in this table were collected in the EEA Event Survey 
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Appendix C. Results of EEA Event Survey and Follow-up Survey 


Table A - 13. Tests of Significance in Knowledge Test Score Differences by Provider— High School ELA 


Scenarios 


Homogeneity 





Variable 

of variance 

T 

DF 

SIG Mean diff ES Interpretation 

Raw score 

.267 

.647 

76 

.520 

.286 .14 Small effect - 

S1S2 






no difference 

NOTE: Data in this table were collected in the EEA Event Survey 

Table A - 

14. Knowledge Test Item Statistics 

— Elementary School Mathematics Scenarios 





% Correct 

PBS Correlation 

Discrimination 

Item 

Scenario 

Number tested 

(difficulty) 

(discrimination) 

Difficulty flag 1 flag 2 

1 

1 


93 

60.2 

.021 

Y 

2 

1 


93 

33.3 

.170 

Y 

3 

1 


93 

58.1 

.482 


4 

1 


93 

72.0 

.291 


5 

1 


93 

24.7 

.082 

Y 

1 

2 


87 

80.5 

.460 


2 

2 


87 

82.8 

.043 

Y 

3 

2 


87 

81.6 

.351 


4 

2 


87 

96.6 

.235 

Y 

5 

2 


87 

94.3 

.436 

Y 

6 

2 


87 

90.8 

.225 

Y 

7 

2 


87 

80.5 

.294 



1 % Correct is < 15% or > 85% 

2 PBS <20 

NOTE: Data in this table were collected in the EEA Event Survey 


Table A - 15. Tests of Significance in Knowledge Test Score Differences by Provider— Elementary School 


Mathematics Scenarios 

Variable 

Homogeneity 
of variance T 

DF 

SIG 

Mean diff 

ES Interpretation 

Raw score 

S1S2 

.307 .380 

83 

.705 

-.120 

.08 Very small 
effect - no 

difference 


NOTE: Data in this table were collected in the EEA Event Survey 


52 I Educator Enhancement Academies Evaluation Study 


Appendix C. Results of EEA Event Survey and Follow-up Survey 


Table A - 16. Knowledge Test Item Statistics— Middle School Mathematics Scenarios 

% Correct PBS Correlation Discrimination 


Item Scenario Number tested (difficulty) (discrimination) Difficulty flag 1 flag 2 


1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

1 

1 

1 

1 

1 

2 

2 

2 

2 

2 

116 

116 

116 

116 

116 

117 

117 

117 

117 

117 

56.9 

31.0 

78.4 

75.0 

74.1 

82.9 

24.8 

90.6 

47.0 

59.8 

.166 

.113 

.056 

.018 

.033 

.464 

.136 

.102 

.480 

.241 

Y 

Y 

Y 

Y 

Y 

Y 

Y Y 

1 % Correct is < 15% or > 85% 





2 PBS <20 






NOTE: Data in this table were collected in the EEA Event Survey 


Table A -17. 

Tests of Significance in Knowledge Test Score Differences by Provider— 

Middle School 


Mathematics Scenarios 





Homogeneity 





Variable 

of variance 

T 

DF 

SIG Mean diff 

ES Interpretation 

Raw score 

.641 

.541 

112 

.589 -.279 

.10 Small effect - 

S1S2_CLEAN 





no difference 

NOTE: Data in this table were collected in the EEA Event Survey 

Table A -18. 

Knowledge Test Item Statistics— High School Mathematics Scenarios 





% Correct 

PBS Correlation 

Discrimination 

Item Scenario Number tested 

(difficulty) 

(discrimination) Difficulty flag 1 flag 2 

1 

1 

95 

69.5 

.209 


2 

1 

95 

46.3 

.261 


3 

1 

95 

89.5 

.145 

Y Y 

4 

1 

95 

70.5 

.208 


5 

1 

95 

54.7 

.225 


1 

2 

96 

74.0 

.462 


2 

2 

96 

37.5 

.016 

Y 

3 

2 

96 

85.4 

.214 

Y 

4 

2 

96 

46.9 

.382 


5 

2 

96 

69.8 

.233 



1 % Correct is < 15% or > 85% 

2 PBS <20 

NOTE: Data in this table were collected in the EEA Event Survey 
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Appendix C. Results of EEA Event Survey and Follow-up Survey 


Table A - 19. Tests of Significance in Knowledge Test Score Differences by Provider— High School 
Mathematics Scenarios 
Homogeneity 


Variable 

of variance 

T 

DF 

SIG 

Mean diff 

ES Interpretation 

Raw score 
S1S2_CLEAN 

.625 

.244 

92 

.808 

.343 

.05 Very small 
effect - no 

difference 


NOTE: Data in this table were collected in the EEA Event Survey 


Table A - 20. The Extent to Which the Educator Enhancement Academies Were Led by Qualified and 


Knowledgeable Trainers by Source of Trainers 


All 


Corwin 


WVDE 


Number 

Mean 

Number 

Mean 

Number 

Mean 

Facilitators 

responses 

response 

responses response 

responses 

response 

Q3.2.1. were knowledgeable about 







the topic 

700 

4.46 

181 

3.74 

519 

4.72 

Q3.2.2. were well organized 

Q3.2.3. presented material clearly, 

700 

4.44 

182 

3.80 

518 

4.66 

effectively 

700 

4.36 

181 

3.55 

519 

4.64 

Q3.2.4. managed discussions well 
Q3.2.5. answered questions 

701 

4.38 

182 

3.63 

519 

4.65 

adequately 

700 

4.37 

181 

3.56 

519 

4.66 

Q3.2.6. modeled desired training 







techniques 

699 

4.27 

182 

3.29 

517 

4.61 

NOTE: Data in this table were collected in the EEA Event Survey 


Table A - 21. Comparison of the Quality of Educator Enhancement Academies to Previous Train-the-Trainer 


Experience(s) by Source of Trainers 


How academy compares to 
previous train-the-trainer 
experience(s) 

All 


Corwin 

WVDE 

Number 

responses 

Percent 

Number 

responses 

Percent 

Number 

responses 

Percent 

All responses 

680 

100.0 

178 

100.0 

502 

100.0 

More useful 

208 

30.6 

21 

11.8 

187 

37.3 

About the same 

186 

27.4 

52 

29.2 

134 

26.7 

Less useful 

77 

11.3 

61 

34.3 

16 

3.2 

No opinion 

23 

3.4 

9 

5.1 

14 

2.8 

No basis to compare 

186 

27.4 

35 

19.7 

151 

30.1 


NOTE: Data in this table were collected in the EEA Event Survey 
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